I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out.

But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to be possible, which then gives amazing leverage to deliver working software in a fraction of the time.

I'd be happier if industry cares more for team productivity - I have witnessed how rewarding "10x" individuals may lead to perverse results on a wider scale, a la Cobra Effect. In one insidious case, our management-enabled, long-tenured "10x" rockstar fixed all the big customer-facing bugs quickly, but would create multiple smaller bugs and regressions for the 1x developers to fix while he moved to the next big problem worthy of his attention. Everyone else ended up being 0.7x - which made the curse of an engineer look even more productive comparatively!

Because he was allowed to break the rules, there was a growing portion of the codebase that only he could work on - while it wasn't Rust, imagine an org has a "No Unsafe Rust" rule that is optional to 1 guy. Organizations ought to be very careful how they measure productivity, and should certainly look beyond first-order metrics.

bingo, well said. Worked on a team like this with a “principal” engineer who’d work very fast with bug-ridden work like this simply because he had the automatic blessing from on high to do whatever he wanted. My unfortunate task was to run along behind him and clean up, which to my credit I think I did a pretty good job at, but of course these types can only very rarely acknowledge/appreciate that.

Eventually he got super insecure/threatened and attempted to push me out along with whoever else he felt was a threat to his fiefdom.

[1] I preserved a copy of it on my (no-advertising or monetization) blog here: https://realmensch.org/2017/08/25/the-parable-of-the-two-pro...

In the early computing era there was a competition. Something like take some input and produce an output. One programmer made a large program in (IIRC) Fortran with complex specifications documentation etc. The other used shell pipes, sort, and a small handful or two of other programs in a pipeline to accomplish the same task in like 10 developer min.

http://catb.org/~esr/writings/unix-koans/ten-thousand.html

"""“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”"""

[1] https://homepages.cwi.nl/~storm/teaching/reader/BentleyEtAl8... [2] https://www.google.com/search?q=knuth+vs+mcilroy

I'm trying to copy-in the program as it might have existed, with some obvious updates to work in today's shells ...

  #!/bin/sh
  tr -cs A-Za-z '
  ' "${2:-/dev/stdin}" |\
  tr A-Z a-z |\
  sort |\
  uniq -c |\
  sort -rn |\
  sed ${1:-100}q

Edited: Removed some errors likely induced by OCR / me not catching that in the initial transcription from the browser view of the file.

[For those who may not follow all the links: Bentley asked Knuth to write a program in Pascal (WEB) to illustrate literate programming—i.e. explaining a long complicated program—and so Knuth wrote a beautiful program with a custom data structure (hash-packed tries). Bentley then asked McIlroy to review the program. In the second half of the review, McIlroy (the inventor of Unix pipes) questioned the problem itself (the idea of writing a program for scratch), and used the opportunity to evangelize Unix and Unix pipes (at the time not widely known or available).]

I drank from the OO koolaid at one point. I was really into building things up using OOD and creating extensible, flexible code to accomplish everything.

And when I showed some code I'd written to my brother, he (rightly) scoffed and said that should have been 2-3 lines of shell script.

And I was enlightened. ;)

Like, I seriously rebuilt my programming philosophy practically from the ground up after that one comment. It's cool having a really smart brother, even if he's younger than me. :)

I revisited earlier thought: communication is a 2 man job, one is to not make an effort to understand while the other explains things poorly. It always manages to never work out.

Periodically I thought about the puzzle and was eventually able to explain it such that people thought it was brilliant ~ tho much to complex to execute.

I thought about it some more, years went by and I eventually managed to make it easy to understand. The response: "If it was that simple someone else would have thought of it." I still find it hilarious decades later.

It pops to mind often when I rewrite some code and it goes from almost unreadable to something simple and elegant. Ah, this must be how someone else would have done it!

This is a good exclamation :D

And it's a poignant story. Thanks for sharing.

― Abraham Lincoln

I have started to follow this 'lately' (for a decade) and it has worked miracles. As for the anxious managers/clients, I keep them updated of the design/documentation/though process, mentioning the risks of the path-not-taken, and that maintain their peace of mind. But this depends heavily on the client and the managers.

Sometimes I'm appreciated, and managers actually realize what they have when I create something for them. Frequently I accomplish borderline miracles and a manager will look at me and say, "OK, what about this other thing?"

My first job out of college, I was working for a company run by a guy who said to me, "Programmers are a dime a dozen."

He also said to me, after I quit, after his client refused to give him any more work unless he guaranteed that I was the lead developer on it, "I can't believe you quit." I simply shrugged and thought, "Maybe you shouldn't have treated me like crap, including not even matching the other offer I got."

I've also made quite a lot of money "Rescuing Small Companies From Code Disasters. (TM)" ;) Yes, that's my catch phrase. So I've seen the messes that teams often create.

The "incompetent" team code description in the story is practically prescient. I've seen the results of exactly that kind of management and team a dozen times. Things that, given the same project description, I could have created in 1/100 the code and with much more overall flexibility. I've literally thrown out entire projects like that and replaced them with the much smaller, tighter, and faster code that does more than the original project.

So all I can say is: Find better teams to work with if you think this is fiction. This resonates with me because it contains industry Truth.

They probably deserve more praise when they do guess correctly but would anyone really know when it happens?

I worked for a company where I almost single handedly built a product that resulted in tens of millions of dollars in sales. I got a nice 'atta boy' for it, but my future ideas were often overridden by someone in management who 'knew better'. After the management changed, I found myself in a downsizing event once I started criticizing them for a lack of innovation.

Which is weird, since so many executives are working in a VC-funded environment, and internal work should be “venture funded” as well.

I don't agree with that, there are a _lot_ of completely crap developers and they get put into positions where even the ones capable of doing so aren't allowed to because it's not on a ticket.

I've seen some thing.

Also, their professional experience is much greater. Sure, their initial jobs at 15 are the occassional weird gig for the uncle/aunt or cousin/nephew but they get picked up by professional firms at 18 and do a job next to their CS studies.

At least, that's how it used to be. Not sure if this is still happening due to the new job environment, but this was the reality from around 2004 to 2018.

For 10x engineers to exist, all it takes is a few examples. To me, everyone is in agreement that they seem to be rare. I point to a public 10x engineer. He'd never say it himself, but my guess is that this person is a 10x engineer [1].

If you disagree, I'm curious how you'd disagree. I'm just a blind man touching a part of the elephant [2]. I do not claim to see the whole picture.

[1] https://bellard.org/ (the person who created JSLinux)

[2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant - if you don't know the parable, it's a fun one!

By the time I joined my first real company at age 21, I was ready to start putting a lot of this stuff into place. I joined a small med device software company which had a great product but really no strong software engineering culture: zero unit tests, using CVS with no branches, release builds were done manually on the COO's workstation, etc.

As literally the most junior person in the company I worked through all these things and convinced my much more senior colleagues that we should start using release branches instead of "hey everybody, please don't check in any new code until we get this release out the door". I wrote automated build scripts mostly for my own benefit, until the COO realized that he didn't have to worry about keeping a dev environment on his machine, now that he didn't code any more. I wrote a junit-inspired unit testing framework for the language we were using (https://en.wikipedia.org/wiki/IDL_(programming_language) - like Matlab but weirder).

Without my work as a "10x junior engineer", the company would have been unable to scale to more than 3 or 4 developers. I got involved in hiring and made sure we were hiring people who were on board with writing tests. We finally turned into a "real" software company 2 or 3 years after I joined.

10x engineers exist. To be fair, it does depend which software engineer you see as "the standard software engineer", but if I take myself as a standard (as an employed software engineer with 5 years of experience), then 10x software engineers exist.

Being a programming prodigy is nice, but I don't think you even really need that.

It's probably especially obvious in the child-prodigy types because we as an industry have a tendency to force people out of IC roles by 40, so the child prodigies are the only ones who have enough time to develop 20 years of experience working directly with code.

The combination of 10-15 years of experience and the energy/time of 20s is very powerful.

in those early years the tasks that you take on are probably above your skillset and fight through it, you're not accountable to anyone.

in a job you're usually hired for what you already know, +/- some margin for more gradual learning , not really that much room for moonshots. the work need to be divided into bit sized parts that you can justify to the higher-ups when needed. you have less room for exploring really non linear paths toward the solution which can be harder to explain but where you learn more.

so in the end this end up sometimes amounting to 10years of experience outside work being more impactful than 10 years at work. ,

I am not sure there is 10x difference, but there is at least 5x difference in performance, in favor of fresh college grad, and they are now working on the more complex tasks too.

The sad part is our hiring is still heavily in "senior engineer with lots of experience" phase, and intership program has been canceled.

At my previous company I had 2 first-job juniors and 4 or 5 interns that were outstanding.

(however the last one was totally terrible and totally killed my hiring credibility hah, but it was a 80% success rate still)

I find that there are too many pretenders, though. I get way too many people with 15 years of experience and ChatGPT resumes that just can't code at all :/

The top-performing experienced developers outshone all the graduates. However, the less effective experienced developers were on par with the graduates, showing no significant difference in performance.

The takeaway for me is that simple anecdotes are not very informative. Over time and with a larger sample size, experienced individuals tend to perform better. Nonetheless, some graduates will also become exceptionally skilled.

Graduates are more cost-effective. Experienced professionals require less oversight. If they need substantial guidance, they don't truly qualify as senior, as opposed to their resume that says, at 25 they have been a CTO for 10 years).

And when we're talking about sports, genetics matter as well (depending on each one)

When we're talking brains, while genetics also matter, assuming normal (whatever that is) brain, the plasticity changes a lot how it operates.

So, the 10 years thing is definitely a big if not the biggest part. In my opinion. Would love to see studies if any exist out there on this

I was in the same math classes as some of my peers for a decade+. Some people were great, some were bad and most were somewhere in between. The kids who were exceptional at 9 were exceptional at 17.

Obviously time matters but genetics play a huge role as well. I have a family friend with 2 adopted kids and 2 biological kids. The adopted ones are average but the biological ones are very smart. Just like their parents.

You probably surf the internet 10x faster than your parents. Yes you've probably had more exposure than them, but you could probably teach them how to do it just as fast. But would they want to learn and would they actually adapt what you taught them?

I became friends with Dave our CTO when I was 5 or 6, we were neighbors. He'd already started coding little games in Basic (this was 1985). Later in our friendship, like when I was maybe 10, I asked him if he could help me learn to code, which he did. After a week or two I had made some progress but compared what I could do to what he was doing and figured "I guess I just started too late, what's the point?".

I found out later that most people didn't start coding till late HS or college! It worked out though - I'm programmer adjacent and have taken care of the business side of our projects through the years :)

No: It's not because they have 10 more years of experience. Read "The Mythical Man Month." That's the book that popularized the concept that some developers were 5-25x faster than others. One of the takeaways was that the speed of a developer was not correlated with experience. At all.

That said, the kind of person who can learn programming at 12 might just be the kind of person who is really good at programming.

I started learning programming concepts at 11-12. I'm not the best programmer I know, but when I started out in the industry at 22 I was working with developers with 10+ years of (real) experience on me...and I was able to come in and improve on their code to an extreme degree. I was completing my projects faster than other senior developers. With less than two years of experience in the industry I was promoted to "senior" developer and put on a project as lead (and sole) developer and my project was the only one to be completed on time, and with no defects. (This is video game industry, so it wasn't exactly a super-simple project; at the time this meant games written 100% in assembly language with all kinds of memory and performance constraints, and a single bug meant Nintendo would reject the image and make you fix the problem. We got our cartridge approved the first time through.)

Some programmers are just faster and more intuitive with programming than others. This shouldn't be a surprise. Some writers are better and faster than others. Some artists are better and faster than others. Some architects are better and faster than others. Some product designers are better and faster than others. It's not all about the number of hours of practice in any of these cases; yes, the best in a field often practices an insane amount. But the very top in each field, despite having similar numbers of hours of practice and experience, can vary in skill by an insane amount. Even some of the best in each field are vastly different in speed: You can have an artist who takes years to paint a single painting, and another who does several per week, but of similar ultimate quality. Humans have different aptitudes. This shouldn't even be controversial.

I do wonder if the "learned programming at 12" has anything to do with it: Most people will only ever be able to speak a language as fluently as a native speaker if they learn it before they're about 13-14 years old. After that the brain (again, for most people; this isn't universal) apparently becomes less flexible. In MRI studies they can actually detect differences between the parts of the brain used to learn a foreign language as an adult vs. as a tween or early teen. So there's a chance that early exposure to the right concepts actually reshapes the brain. But that's just conjecture mixed with my intuition of the situation: When I observe "normal" developers program, it really feels like I'm a native speaker and they're trying to convert between an alien way of thinking about a problem into a foreign language they're not that familiar with.

AND...there may not be a need to explicitly PROGRAM before you're 15 to be good at it as an adult. There are video games that exercise similar brain regions that could substitute for actual programming experience. AND I may be 100% wrong. Would be good for someone to fund some studies.

I started programming at age 7 and it's true that the way code forms in my head feels similar to the way words form when I'm writing or speaking in English. In the same way that I don't stop and consciously figure out whether to use the past or present tense while I'm talking, I usually don't consciously think about, say, what kind of looping construct I'm about to use; it's just the natural-feeling way to express the idea I'm trying to convey. The idea itself is kind of already in the form of mental code in the same way that my thoughts are kind of already in English if I'm speaking.

But... maybe that's how it is for everyone, even people who learned later? I only know how it is in my own head.

I always thought that people who start at 12 and keep at it are good because they really love it.I see people who struggle a lot with learning, and it's because they hate it but are doing it for other reasons.

It's easy to bog down even the best Nx engineers if you keep them occupied with endless bullshit tasks, meetings, (ever) changing timelines, and all that.

Kind of like having a professional driver drive a sportscar through a racetrack, versus the streets of Boston.

"The truest 10x engineer I ever encountered was..."

In the end it doesn't matter, whole team could be laid off at once.

It's not about the hours.

Productivity has no inherent value - like efficiency and perfection, it is necessarily of something else. Its value is entirely derived.

What we do know is that the worst engineers provide negative productivity. If 1x is the worst engineer, then let's for the sake of discussion denote x as -1 in order for the product to be negative. Except that means the 10x engineer provides -10 productivity, actually making them the worst engineer. Therein lies a conflict.

What we also know is that best engineer has positive productivity, so that means the multiplicand must always be positive. Which means that it is the multiplier that must go negative, meaning that a -1x and maybe even a -10x engineer exists.

Throw into the mix the fact that productivity is hard to measure as soon as more than one person works on something and that doesn't even begin to consider the economical aspects of software.

And even when ignoring this point, there's that pesky short-term vs long-term thing.

Also, how do you define the term "productivity"? I was assuming that you mean somethint along the lines of (indirect, if employed) monetary output.

The 10x developer originated from a study that measured performance. The 10x developer being able to do a task in a 10th of the time is quite conceivable and reflects what the study found. I'm sure we've all seen a developer take 10 hours to do a job that would take another developer just 1 hour. Nobody is doing it in negative hours, so the math works.

But performance is not the same as productivity.

Performance. That is what the study that found a 10x performance difference observed. There is no mention of productivity in the study. If anyone has tried to study productivity, they most certainly have not come up with a 10x moniker. It seems productivity was mentioned in this thread only because it also happens to start with the letter 'p' and someone got confused.

Of course, that doesn't actually matter as the original study found a 10x difference in measuring performance, not productivity. There is nothing out there to suggest that some developers are 10x more productive outside of those who mixed up their p words. We're not actually talking about productivity; that was just a mistake. If one were to study productivity, I expect they would find that some engineers are many orders of magnitude more productive than the least productive engineers.

I reject that assertion.

> performance, not productivity

You keep saying this like it's a slam dunk refutation, but performance and productivity are highly related.

Because you don't believe the worst developer contributed anything to GPT? Sure, in reality that's no doubt true, but it was only ever meant to be illustrative.

> but performance and productivity are highly related.

Not in any meaningful way. The study found that the fastest developer can perform a set of defined tasks in a 10th of the time of the slowest developer. That is what the 10x developer refers to. But being fast doesn't mean being productive.

Come to my backyard and we will each dig a hole of equal size. Let's assume that you can dig the hole in a 10th of the time I can – that you are the 10x hole digger. But, no matter how fast you are, neither of us will be productive.

You, on the other hand, are claiming that 10x engineers are 10 times more productive than the worst engineers. Completing a programming challenge in a 10th of the time is not the same as being 10 times more productive, and obviously your usage can't be an explanation, even as one you made up on the spot, as the math doesn't add up.

That study also does not exist in isolation:

https://www.construx.com/blog/the-origins-of-10x-how-valid-i...

Because it was? Do you have a better way to repeatedly test performance? And yes, the study's intent was to look at performance, not productivity. It's even right in the title. Not sure where you dreamed up the latter.

https://www.construx.com/blog/the-origins-of-10x-how-valid-i...

3x bonus multiplies (Redefines Expectations) are extremely uncommon. Level increases certainly help but like, L7 only makes ~3x what L5 does -- not 10x. And there are few L7s and very few L8+.

They work best on their own projects with nobody else in their way, no colleagues, no managers, but that's not most jobs. Once you're part of a team, you can't do too much work yourself no matter how good you are, as inevitably the other slower/weaker team members will slow you down as you'll fight dealing with the issues they introduce into the project or the issues from management, so every team moves at the speed of the lowest common denominator no matter their rockstars.

Yeah but this isn't something scalable that can happen regularly as part of your job description. Like most jobs/companies don't have so many low hanging fruits to pick that someone can speed of build by orders of magnitude on a weekly basis. It's usually a one time thing. And one time things don't usually make you a 10x dev. Maybe you just got lucky once to see something others missed.

And often times at big places most people know where the low hanging fruits are and can fix them, but management, release schedules and tech debt are perpetually in the way.

IMHO what makes you a 10x dev is you always know how to unblock people no matter the issue so that the project is constantly smooth saling, not chasing orders of magnitude improvements unicorns.

I personally think that some people can provide “10x” (arbitrary) the value on occasion, like the low hanging fruit you said. I also believe some people are slightly more skilled than others, and get more results out of their work. That said, there are so many ways for somebody to have an impact that doesn’t have to immediate, that I find the term itself too prevalent.

Agreed, there is too much effort going into the "superstars" theme, but there are definitely people who get 10x done in the same time as others.

Some of those people not only get things done much quicker, but they also get it done with better quality than an amateur, with less mistakes, throwing away less material, sometimes with more safety.

This is definitely more than 10x better. And there are some real hacks doing those kinds of jobs. I find programming to be not different than that.

You and I have worked at very different organizations. Everywhere I've been has had insane levels of inefficiency in literally every process.

In processes yes, not in code, and solo 10x devs alone can't fix broken processes as those are a the effect of broken management and engineering culture.

People know where the inefficiencies are, but management doesn't care.

Over time this produces funny effects, like super-big 20 point task done in few days because wrong person started working on it.

"Hmmm, there's another way to accomplish this" being the 10x. Doing things faster is not it.

(And absolutely not brute-force grinding themselves away)

https://manuel.kiessling.net/2011/04/07/why-developing-witho...

How fast would you drive a car if I gave you the keys and told you „everything works perfectly fine, however the brakes have been removed“?

It is always very useful as an engineer to log your time, what are you working on "right now" and is it "new work" , "maintenance work", or "fixing work." Then for each log entry that isn't "new work" thinking about what you could have done that would have caught that problem before it was committed to the code base.

I find it is much better to evaluate engineers based on how often they are solving the same problem that they had before vs creating new stuff. That ratio, for me, is the essence of the Nx engineer (for 0.1

The point that Wilson makes that having infrastructure/tools that push that ratio further from "repair" work to "new work" is hugely empowering to an organization.

At other times I’ll be assigned projects with regular POs, Architects and business employees who barely know what it is they are doing themselves, with poorly defined tasks and all sorts of bureaucratic nonsense “agile” process methods and well spend forever delivering nothing.

So sometimes I’m a 50x developer delivering business altering changes. At other times I’m a useless cog in a sea of pseudo workers. I don’t particularly care, I get paid, but if management actually knew what was going on, and how to change it… well…

I think the reluctance to invest effort in something that will give devs super-powers 6 months in the future is why we don't get all those 10x devs.

I agree with the first part of your statement, but what really happens to such people?

In my experience (sample size greater than one), they receive some kudos, but remain underpaid, never promoted, and are given more work under tight deadlines. At least until some of them are laid off along with lower performers.

But for those who say that hard things are impossible, they seem to get along just fine. They merely declare such things as out-of-scope or lie about their roadmap.

100% agree, I've seen plenty of the best of the best get treated like trash and laid off at first sight of trouble on the horizon

I don't think that tracks. Smart, productive, hard working people don't work 9-5. They work every hour they can, breaking only when they have pushed themselves to the limit. The limit can be hit at any hour. There is no magical property of the universe that gives people unlimited stamina during the hours of 9-5.

> Nobody would call this person "10x".

I'm not sure they would call anyone that, to be fair. A "10x developer" who also puts in 8 hours alongside the 1x developers isn't a 10x developer, he would be called a sucker.

Another flaw in his thinking: brain cycles and sub-conscious processing.

I'm in the middle of a hard problem right now. I ran out of ideas, and opened HN about half an hour ago. In that time, without "trying", I've had two new ideas - one sent me back to my notes, which revealed that my original thinking was flawed; the second sent me to documentation, which suggested a new route to pursue. I'm digesting the implications of that while I write this.

Beating my head against the problem directly for thirty minutes would have been less productive. (Though if I wasn't WFH I would have, and also been miserable, and learned less about the industry than I have from this thread. So there's that.)

I'm far from a 10x anything, but I don't have the only brain which works this way.

Lays the foundation (get it?) for who the people are and what they've built.

Then explains how the current thing they are building is a result of the previous thing. It feels that they actually want this problem solved for everyone because they have experienced how good the solution feels.

Then tells us about the teams (pretty big names with complex systems) that have already used it.

All of these wrapped in good writing that appeals to developers/founders. Landing page is great too!

It would be nice to see some actual use cases and examples.

Instead, the writer just name-dropped a few big companies and claimed to have a revolutionary product that works magically. Then include the typical buzzwords like '10x programmer' and 'stealth mode'. The latter doesn't make sense because they also name-drop clients.

Also, stealth mode just means your company isn’t public, you can still have clients.

Having that context puts the post in a much better perspective. It's definitely an introduction post (the company has been developing this in stealth mode for the past few years), but it is most certainly _not_ a marketing post. These people developed extremely novel testing techniques for FoundationDB and are now generalizing them to work with any containerized application.

It's a big deal.

Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, invented by functional programmers. There's a thing, called network simulation, invented by distributed systems people. There's a thing, called rare-event simulation, invented by physicists (!). But if you squint, all of these things are really the same kind of thing, which we call "autonomous testing". It's where you express high-level properties of your system, and have the computer do the grunt work to see if they're true. Antithesis is our attempt to take the best ideas from each of these fields, and turn them into something really usable for the vast majority of software.

We believe the two fundamental problems preventing widespread adoption of autonomous testing are: (1) most software is non-deterministic, but non-determinism breaks the core feedback loop that guides things like coverage-guided fuzzing. (2) the state space you're searching is inconceivably vast, and the search problem in full generality is insolubly hard. Antithesis tries to address both of these problems.

So... is it fuzzing? Sort of, except you can apply it to whole interacting networked systems, not just standalone parsers and libraries. Is it property-based testing? Sort of, except you can express properties that require a "global" view of the entire state space traversed by the system, which could never be locally asserted in code. Is it fault injection or chaos testing? Sort of, except that it can use the techniques of coverage guided fuzzing to get deep into the nooks and crannies of your software, and determinism to ensure that every bug is replayable, no matter how weird it is.

It's hard to explain, because it's hard to wrap your arms around the whole thing. But our other big goal is to make all of this easy to understand and easy to use. In some ways, that's proved to be even harder than the very hard technological problems we've faced. But we're excited and up for it, and we think the payoff could be big for our whole industry.

Your feedback about what's explained well and what's explained poorly is an important signal for us in this third very hard task. Please keep giving it to us!

[0] https://www.youtube.com/watch?v=4fFDFbi3toc [1] https://thestrangeloop.com/2014/testing-distributed-systems-...

It's definitely true though that a big problem with backend is that you can't easily treat it as a whole system for test purposes.

or starting a bidding war

Would it work for debugging, say, Notepad on Windows?

Disclosure: Antithesis co-founder.

Doesn't Antithesis rely on the fact that software is always deterministic? Reproducibility appears to be its top selling feature – something that wouldn't be possible if software were non-deterministic.

* Offer only good for x86-64 software that runs on Linux whose dependencies you can install locally or mock. The first two restrictions we will probably relax someday.

However, if we pretend that software was somehow able to be non-deterministic, it would be able to evade your deterministic computer. But since software is always deterministic, you just have to guarantee determinism in the inputs.

>But since software is always deterministic, you just have to guarantee determinism in the inputs.

This is technically correct, but that's a very load-bearing "just". A lot of things would have to count as inputs. Think about execution time, for example. CPUs don't execute at the same speed all the time because of automatic throttling. Network packets have different flight times. Threads and processes get scheduled a little differently. In distributed/concurrent systems, all this matters. If you run the same workload twice, observable events will happen at different times and in different orders because of tiny deviations in initial conditions.

So yes, if you consider the time it takes to run every single machine instruction as an "input", then software is deterministic given the same inputs. But in the real world that's not actionable. Even if you had all those inputs, how are you going to pass them in? For all intents and purposes most software execution is non-deterministic.

The Antithesis simulation is deterministic in this way though. It is in charge of how long everything takes in "simulated time", right down to the running times of individual CPU instructions. Everything observable from within the simulation happens the exact same way, every time. You can compare a memory dump at the same (simulated) instant across two different runs and they will be bit-for-bit identical.

Sure. A good example. Execution time – more accurately, execution speed – isn't a property of software. For example, as you point out yourself, you can alter the execution speed without altering the software. It is, indeed, an input.

> Even if you had all those inputs, how are you going to pass them in?

Well, we know how to pass them in non-deterministically. That's how software is able to do anything.

Perhaps one could create a simulated environment that is able to control all the inputs? In fact, I'm told there is a company known as Antithesis working on exactly that.

Is the challenge here the same as with digital simulations of electronic circuits? That is, at the end of the day analog physics becomes confounding? Or are you doing deterministic simulation of random RF noise as well?

It's hard to understand these complex concepts via language alone.

Diagrams would be a huge help to understand how this system of testing works compared to existing testing concepts

So Jefferson just says, let's take your I/O-ful process, split it a message-passing actor model, and monitor all the messages going in and coming out. The messages coming out, they won't necessarily do what they're supposed to do yet, they'll just be recorded with a plus sign and a virtual timestamp, and by assumption eventually you'll block on some response. So we have a bunch of recorded message timestamps coming in, we have your recorded messages going out.

Well, there's a problem here, which is that if we have multiple actors we may discover that their timestamps have traveled out-of-order. You sent some message at t=532 but someone actually sent you a message at t=231 that you might have selected instead of whatever you actually selected to send the t=532 message. (For instance in the OS case, they might have literally sent a SIGKILL to your process and you might not have sent anything after that.) That's what the plus sign is for, indirectly: we can restart your process from either a known synchronization state or else from the very beginning, we know all of its inputs during its first run so we have "determinized" it up past t=231 to see what it does now. Now, it sends a new message at say t=373. So we use the opposite of +, the minus sign, to send to all the other processes the "undo" message for their t=532 message, this removes it from their message buffer: that will never be sent to them. And if they haven't hit that timestamp in their personal processing yet, no further action is needed, otherwise we need to roll them back too. Doing so you determinize the whole networked cluster.

The only other really modern implementation of these older ideas that I remember seeing was Haxl[2], a Haskell library which does something similar but rather than using a virtual time coordinate, it just uses a process-local cache: when you request any I/O, it first fetches from the cache if possible and then if that's not possible it goes out, fetches the data, and then caches it. As a result you can just offer someone a pre-populated cache which, with these recorded inputs, will regenerate the offending stack trace deterministically.

1: https://dl.acm.org/doi/10.1145/3916.3988

2: https://github.com/facebook/Haxl

I haven't heard of deterministic testing before. Nor have I heard of FoundationDB or the related things. And I went from knowing zero things about them to getting impressed and interested. This led me to go into their docs, blogs, landing page, etc. to know more.

The most pernicious, hard-to-find bugs that I've come across have all been around the business logic of an application, rather than it hitting into an error state. I'm thinking of the category where you have something like "a database is currently reporting a completed transaction against a customer, but no completed purchase item, how should it be displayed on the customer recent transactions page?". Implementing something where "a thing will appear and not crash" in those cases is one thing, but making sure that it actually makes sense as a choice given all the context of everyone elses choices everywhere else in the stack is a lot harder.

Or to take a database, something along the lines of "our query planner produces a really suboptimal plan in this edge-case".

Neither of those types of problems could ever be automatically detected, because they aren't issues of the programming reaching an error state- the issue is figuring out in the first place what "correct" actually is for you application.

Maybe I'm setting the bar too high for what a "bug" is, but I guess my point is, its one thing to fantasize about having zero bugs, its another to build software in the real world. I probably still settle for 0 run time errors though to be fair. . .

So in normal cases this would reek of someone being arrogant / overconfident, but here they really have gotten very close to zero bugs.

To be clear, I am not saying that you can't define all inputs and outputs of a "complete SaaS product offering stack", because you likely could, though if it's already been built by someone that doesn't have these things in mind, then it's a different problem space to find bugs.

As someone who has spent the last 15 years championing quality strategy for companies and training folks of varying roles on how to properly assess risk, it does indeed feel like this has a more narrow scope of "bug" as a definition, in the sort of way that a developer could try to claim that robust unit tests would catch "any" bugs, or even most of them. The types of risk to a software's quality have larger surface areas than at that level.

Also, when properties are difficult to think of, that often means that a model of the behavior might be more appropriate to test against, e.g. https://concerningquality.com/model-based-testing/. It would take a bit of design work to get this to play nicely with the Antithesis approach, but it's definitely doable.

The point was more that, I don't see how this testing approach (at the level that it functions) would catch all of the bugs that I have seen in my career, and so to say "all of the bugs" or even "most of the bugs" is definitely a stretch.

This is certainly useful, just like unit tests, assertions, etc are all very useful. It's just not the whole picture of "bugs".

They should clarify that this is focused on functional logic bugs though, I agree with that.

I’d settle for decent heap debugging.

Issues around business logic are not failures of the system, the system worked to spec, the spec was not comprehensive enough and now we iterate.

Because I'd call that a bug. A spec bug, but a bug. It's no feature request to make the code based on the newer page delete thing2 rather than thing1, it's fixing a defect

Instead of changing one word in the code that defines the access level required for this GET call, it gets triaged as not being a bug, put on a backlog, and we never heard from it again obviously

We pay for this shit

Correctness bug: it didn’t do what the spec says it should do.

Fitness for purpose bug: it does what the spec says to do, but, with better knowledge, the spec isn’t what you actually want.

Edit: looks like this maps, respectively, to failing verification and failing validation. https://news.ycombinator.com/item?id=39359673

Edit2: My earlier comment on the different things that get called "bugs", before I was aware of this terminology: https://news.ycombinator.com/item?id=22259973

Verification is "does this thing do what I asked it to do".

Validation is "did I ask it to do the right thing".

verification refers to "is this dataset clean?" or the more precise "does this dataset confirm my assumptions about what a what a correct dataset should be given its focus"

validation refers to "can it answer my questions?" or the more rigorous "can I test my hypotheses against this dataset?"

So I find this interesting (but in hindsight unsurprising) that similar definitions are used in other fields. Would you have a source for your defintions?

E.g. see Wikipedia: https://en.m.wikipedia.org/wiki/Verification_and_validation

The actual testing value is the difference between the cost of writing and maintaining the code, and the cost of writing and maintaining the spec.

If the spec is similar in complexity to the code itself, then bugs in the spec are just as likely as bugs in the code, thus verification to spec has gained you nothing (and probably cost you a lot).

So yes, I totally agree with GP and would actually go further: a phrase like "we found all the bugs in the database" is nonsense and makes the article less credible.

I agree that suboptimal query planning would be a database-layer bug, a defect which could easily be missed by the bug-testing framework.

I myself, prefer to work with the people who will actually use what I build - get a better product hat way.

Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches all sources of non-determinism (including dependencies that call out to the OS). It's pretty seamless.

The author of this article isn't joking when they say that the startup cost of this effort is monumental. Dealing with every possible source of non-determinism, re-writing services to be testable/sans-IO [2], etc. takes a lot of engineering effort.

Once the system is in place though, it's hard to describe just how confident you feel in your code. Combined with tools like quickcheck [3], you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.

This kind of testing is an incredibly powerful tool to have in your toolbelt, if you have the patience and fortitude to invest in it.

As for Antithesis itself, it looks very very cool. Bringing the deterministic testing down the stack to below the OS is awesome. Should make it possible to test entire systems without wiring up a harness manually every time. Can’t wait to try it out!

[0]: https://sled.rs/simulation.html

[1]: https://github.com/madsim-rs/madsim?tab=readme-ov-file#madsi...

[2]: https://sans-io.readthedocs.io/

[3]: https://github.com/BurntSushi/quickcheck?tab=readme-ov-file#...

Are there public examples of what such a re-write looks like?

Also, are you working at a rust shop that's developing this way?

Final Note, TigerBeetle is another product that was written this way.

I gotta say, that is some next-level commitment to writing a good database.

Disclosure: Antithesis co-founder here.

- the company that developed Madsim (RisingWave) [0] [1] is tries hardest to eliminate non-determinism with the broadest scope (stubbing out syscalls, etc.)

- sled [2] itself has an interesting combo of deterministic tests combined with quickcheck+failpoints test case auto-discovery

- Dropbox [3] uses a similar approach but they talk about it a bit more abstractly.

Sans-IO is more documented in Python [4], but str0m [5] and quinn-proto [6] are the best examples in Rust I’m aware of. Note that sans-IO is orthogonal to deterministic test frameworks, but it composes well with them.

With the disclaimer that anything I comment on this site is my opinion alone, and does not reflect the company I work at —— I do work at a rust shop that has utilized these techniques on some projects.

TigerBeetle is an amazing example and I’ve looked at it before! They are really the best example of this approach outside of FoundationDB I think.

[0]: https://risingwave.com/blog/deterministic-simulation-a-new-e...

[1]: https://risingwave.com/blog/applying-deterministic-simulatio...

[2]: https://dropbox.tech/infrastructure/-testing-our-new-sync-en...

[3]: https://github.com/spacejam/sled

[4]: https://fractalideas.com/blog/sans-io-when-rubber-meets-road...

[5]: https://github.com/algesten/str0m

[6]: https://docs.rs/quinn-proto/0.10.6/quinn_proto/struct.Connec...

As cool as all this is, I can't stop but wonder how often the culture of micro-services and distributed computing is ill advised. So much complexity I've seen in such systems boils down to calling a "function" is: async, depends on the OS, is executed at some point or never, always returns a bunch of strings that need to be parsed to re-enter the static type system, which comes with its own set of failure modes. This makes the seemingly simple task of abstracting logic into a named component, aka a function, extremely complex. You don't need to test for any of the subtle failures you mentioned if you leave the logic inside the same process and just call a function. I know monoliths aren't always a good idea or fit, at the same time I'm highly septical whether the current prevalence of service based software architectures is justified and pays off.

You can't get away from distributed computing, unless you get away from computing. A modern computer isn't a single unit, it's a system of computers talking to each other. Even if you go back a long time, you'll find many computers or proto-computers talking to each other, but with a lot stricter timings, as the computers are less flexible.

If you save a file to a disk, you're really asking the OS (somehow) to send a message to the computer on the storage device, asking it to store your data, and it will respond with success or failure and it might also write the data. (sometimes it will tell your os success and then proceed to throw the data away, which is always fun)

That said, keeping things together where it makes sense, is definitely a good thing.

> Programming in this state is like living life surrounded by a force field that protects you from all harm. [...] We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it _had no bugs_.

Being able to make that statement and back it by evidence must be indeed a cool thing.

pp. 65-66:

> The longer I have computed, the less I seem to use Numerical Software Packages. In an ideal world this would be crazy; maybe it is even a little bit crazy today. But I've been bitten too often by bugs in those Packages. For me, it is simply too frustrating to be sidetracked while solving my own problem by the need to debug somebody else's software. So, except for linear algebra packages, I usually roll my own. It's inefficient, I suppose, but my nerves are calmer.

> The most troubling aspect of using Numerical Software Packages, however, is not their occasional goofs, but rather the way the packages inevitably hide deficiencies in a problem's formulation. We can dump a set of equations into a solver and it will usually give back a solution without complaint - even if the equations are quite poorly conditioned or have an unsuspected singularity that is distorting the answers from physical reality. Or it may give us an alternative solution that we failed to anticipate. The package helps us ignore these possibilities - or even to detect their occurrence if the execution is buried inside a larger program. Given our capacity for error-blindness, software that actually hides our errors from us is a questionable form of progress.

> And if we do detect suspicious behavior, we really can't dig into the package to find our troubles. We will simply have to reprogram the problem ourselves. We would have been better off doing so from the beginning - with a good chance that the immersion into the problem's reality would have dispelled the logical confusions before ever getting to the machine.

I suppose whether to do this depends on how rigorous one is, how rigorous certain dependencies are, and how much time one has. I'm not going to be writing my own database (too complicated, multiple well-tested options available) but if I only use a subset of the functionality of a smaller package that isn't tested well, rolling my own could make sense.

I was a co-founder of both FoundationDB and Antithesis.

One example that comes to mind where I want to roll my own thing (and am in the process of doing so) is replacing our ci/cd usage of jenkins that is solely for running qa automation tests against PR's on github. Jenkins does way way more than we need. We just need github PR interaction/webhook, secure credentials management, and spawning ecs tasks on aws...

Every time I force myself to update our jenkins instance, I buckle up because there is probably some random plugin, or jenkins agent thing, or ... SOMETHING that will break and require me to spend time tracking down what broke and why. 100% surface area for issues, whilst we use

I do not make the claim my spec has no bugs.

If you are writing software, it is almost always trying to accomplish a goal outside of itself. It is trying to solve a problem for someone, and how that problem can or should be solved is rarely perfectly clear.

The spec is supposed to map to a real world problem, and there is never going to be a way to formalize that mapping.

Formal verification is incredibly, amazingly good if you achieve it, but it's not the same as "perfect".

You can claim that your spec doesn't violate some invariants in a finite number of steps, you can't claim that the spec contains all the invariants the real system must have and that it doesn't violate them in number of steps + 1.

1. It's a brilliant idea that came at the right time. It feels like people are finally losing patience with flaky software, see developer sentiment on: fuzzers, static typing, memory safety, standardized protocols, containers, etc.

2. It's meant to be niche. $2 per hour per CPU (or $7000 per year per CPU if reserved), no free tier for hobby or FOSS, and the only way to try/buy is to contact them. Ouch. It's a valid business model, I'm just sad it's not going for maximum positive impact.

3. Kudos for the high quality writing and documentation, and I absolutely love that the docs include things like (emphasis in original):

> If a bug is found in production, or by your customers, you should demand an explanation from us.

That's exactly how you buy developer goodwill. Reminds me of Mullvad, who I still recommend to people even after they dropped the ball on me.

Disclosure: Antithesis co-founder.

This is the sort of thing that, if it takes off, will start affecting the entire software world. Hardware will start adding features to support it. In 30 years this may simply be how computing works. But the pioneers need to recover the costs of the arrows they got stuck with before it can really spread out. Don't look at this an event, but as the beginning of a process.

Heck, most software in production today barely has unit tests.

I actually got to follow their bug tracking process on an issue they identified in Apache Spark streaming - going off of the docs, they managed to identify a subtle and insidious correctness error in a common operation that would've caused headaches in low visibility edge case for years at that point. In the end the docs were incorrect, but after that showing I cannot imagine how critical tools like Antithesis will be inside companies building distributed systems.

I hope we get some blog posts that dig into the technical weeds soon, I'd love to hear what brought them to their current approach.

Ok but, what actually IS it?

It seems like it is a cloud service that will run integration tests. I have to figure out how to deploy to this special environment and I still have to write those integration tests using special libraries.

But even after all that integration refactoring, how is this supposed to help me find actual bugs that I wouldn't already have found in my own environment with my own integration tests?

That said, Antithesis doesn't require you to write manual tests, integration or otherwise. It requires your software system to be packaged in containers, which is fairly straightforward, and then requires a workload to be written which will emulate the normal functioning of the software system. So for example an e-commerce store would have product views, cart adds, checkouts, etc.

With this, Antithesis can start testing (running your workload, varying inputs, injecting faults, etc) the software and looking for violations of test properties. There are many (60+) test properties that come "out of the box" such as crashes, out of memory, etc. You can (and should) also define custom properties that are unique to your system, as this will surface more problems.

As your tests run, violations of test properties are reported, with lots of useful debug information included. Test runs that are particularly interesting can have a lot of extra analysis done, due to our ability to "rewind" and change inputs, get artifacts, add logging, etc.

I don't mean to poke holes but I'm having trouble seeing the value add here.

If I have to deploy to some new environment anyways and I have to tailor the "Workloads" anyways why would I pay extra for vendor lock-in?

The type of devious bug this is promising to find would be something like:

"The DB silently drops timezone from Dates because of the column type. This results in unexpected data being returned for users in different timezones from the server"

I just don't see how repeatably calling the API with an expanding set of random inputs helps find something like that.

But that's all we know so far. I'm assuming there'll be some sort of fuzz testing, and static analysis or some defining actions that your software can perform.

Honestly it sounds a lot like it has a lot of crossover with what the Vale language is trying to solve: https://vale.dev/, but focused on trying to get existing software to that state instead of creating a new language to make new software already be at that state by default.

This means they use their hypervisor to change random seeds, make http requests fail or take too long, break connections between servers, change the order of server responses, and all sorts of wild things you don’t usually control, but that happen in the real world. Then they compare the expected workload responses and figure out which conditions break your systems.

That’s why they sell yearly contracts - you’re supposed to pay them to keep your workloads running continuously all year to try all sorts of different combinations of failures.