"This is entirely server-side UA sniffing going wrong. You get an empty HTML doc, only a doctype, with a Firefox Android UA. You can reproduce this with curl,

    $ curl -H "User-Agent: Mozilla/5.0 (Android 10; Mobile; rv:123.0) Gecko/123.0 Firefox/123.0" https://www.google.com
    %

Whatever UA string Firefox Mobile sends in desktop mode returned fine.

https://camo.githubusercontent.com/16a554571b3cea973d2b73464...

If you're talking about Google's homepage, the answer is "a lot". You can check for yourself - go to google.com, select "view source" and compare the amount of Closure-compiled JavaScript against HTML markup.

- We're on the verge of another browser monopoly, cheered on by developers embracing the single controlling vendor;

- We already have sites declaring that they "work best in Chrome" when what they really mean is "we only bothered to test in Chrome".

- People are not only using UA sniffing with inevitable disastrous results, they're proclaiming loudly that it's both necessary and "the best" solution.

- The amount of unnecessary JavaScript is truly gargantuan, because how else are you going to pad your resume?

I mean really what's next?

Are we going to start adopting image slice layouts again because browsers gained machine vision capabilities?

It's not amateurish to have problems. It is amateurish (or malicious) to have problems caused by this specific class of issue.

If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

(Submitted title was "Google breaks search for Firefox users because of bad UA string sniffing")

Disclosure: I work at Google but not on this. This was linked from the Bugzilla bug.

There's a feature important to Middle Manager 32456. You can't just revert it for a not-Google browser. That's just a no-go.

So a fix needs to be developed, QA'd, rolled. I presume it's going to be an out-of-schedule roll so it'll probably involve some bureaucracy. (One of my clients is a few thousand people public company and even there a hotfix requires QA lead approval.)

Nothing ever is simple.

OTOH, it's amazing that apparently they don't have UI tests for FF mobile.

Why? They don't have UI tests for Opera on the Nintendo Wii either, and at this point I bet the install base for Wii-Opera is still larger than the install base for FF mobile.

TBH, when I was there it surprised me that Google didn't have a dedicated hardware test-bench room with rows upon rows of browser deployments that every UI change needed to be burn-in tested on, but... They don't. They never did. In general, their strategy is to be nimble and deploy rapidly (with the expectation they can roll back rapidly). In that context, it actually makes sense why they don't have that warehouse of test-bench installations... They'd slow-down rollbacks as well as rollouts.

A handful of projects have dedicated testing targets. They're driven mostly by the ideology of individual Googlers (some people really like Firefox) and a handful of high-value users that have specific installs Google isn't interested in pissing off. Since they do very little (relatively speaking) B2B business, that's a very short list of names.

(From the status page linked above.)

That sounds pretty far from being nimble and deploying rapidly. Not really a knock on the people doing the work — doing stuff in high stress sucks. But it's clear that they're intentionally deprioritizing a competitor.

For two, that "next update in 12 hours" is user comms. For me, at least, google.com works fine both on curl and my browser. That's a fairly normal cadence for big companies.

On the larger point about "nimble and deploying rapidly", the people who generally brag about "being nimble and deploying rapidly" almost never serve an even 1/100th the audience the size of Google.com does, and it's really questionable if, at that scale, you actually want to risk global regressions even on trivial bugs.

So I don't know what that user is talking about, and I agree with you that they are obviously not that.

That approach may be antithetical to the modern startup engineer frantic to prove their stock's hypothetical worth to their investors, unconcerned about trivial revenue loss from frontpage issues because of whatever latest node.js drama nuked their continuously deployed website. But the fact that "the landing search page is broken for 1% of users in a rare but public use case" is news at all is because Google's approach for search sets our expectations that this won't happen.

I remember an internal mantra in Google along the lines of "if you break something in production, roll back first, ask questions about how it happened later, even if you think it's a simple fix". It feels telling that this is not what is happening in this circumstance.

> if you break something in production, roll back first, ask questions about how it happened later, even if you think it's a simple fix

I don't know that they have enough information on the breakage to know when to rollback to.

But it's also not on the order of quarters, is what I mean.

Relatively speaking, you can still undo the change to the front page faster than you can, say, roll out a new version of a desktop application, especially if the change fixes an active fire.

From Google's standpoint, this issue may be considered a "new" bug, which means they need to conduct an investigation and address it. Consequently, a rollback is not a viable solution for them.

https://news.ycombinator.com/item?id=38349357

This keeps happening over and over for Google at a rate far in excess of other major tech companies so we know they don’t make an effort to test with Firefox, and it’s why I’d like regulators to force anyone making a browser used by more than, say, 100k people to be obligated to test in other browsers on every site their larger organization operates and have a maximum time to patch any compatibility or performance issue which isn’t due to a browser not implementing a W3C standard. The goal would be to cover things like YouTube using the Chrome web component API for ages after the standard shipped and serving a slow polyfill to non-Chrome browsers.

FWIW, over my time at Google (2012-2022) we had massively better testing on Firefox than Safari, because you can run Firefox on Linux on arbitrary hardware while Safari requires MacOS which requires Apple hardware and is automation-hostile. We did eventually get some tests on Safari, but they were slower and flakier.

(Not defending; trying to give background.)

I appreciate that - it’s been really frustrating knowing how many good technical staff Google has and seeing things just not get fixed. Our GCP reps seemed resigned to this happening but didn’t feel they had leverage to get attention on the product side, which made me feel bad for them.

If it looks like a banana, shaped like a banana, feels like a banana, it’s probably a banana (or a plantain).

> "Over and over. Oops. Another accident. We'll fix it soon. We want the same things. We're on the same team. There were dozens of oopses. Hundreds maybe?"

> "I'm all for 'don't attribute to malice what can be explained by incompetence' but I don't believe Google is that incompetent. I think they were running out the clock. We lost users during every oops. And we spent effort and frustration every clock tick on that instead of improving our product. We got outfoxed for a while and by the time we started calling it what it was, a lot of damage had been done," Nightingale said.

I tend to believe him

To prevent the market share from going up? That's the logical thing to do after spending millions developing and marketing your browser to the number 1 spot.

Why would they waste that money and effort by playing nice now and letting the market share go back up?

> Get a faster computer

or... and keep up with me here. Google has been known to do A/B testing on users. Just because your part of "B" group that doesn't see it doesn't mean that it's not happening. It only means Google is artificially putting their thumb on the scale and I take serious issue with people handwaving issues with "get a better computer" Nothing is wrong with my computer and only Youtube has this issue, any other video platform does not do this so why is it only Youtube?

Yeah my m2 max is a total fucking slouch, just barely better than a 486.

Not to mention the fact that this is google.com, not some wild feat of frontend engineering. They shouldn't be serving any JS that wouldn't run on IE, forget anything close to a modern browser.

This is entirely server-side UA sniffing going wrong. You get an empty HTML doc, only a doctype, with a Firefox Android UA. You can reproduce this with curl and it seems that this affects all UA strings with versions >= 65.

> Warning: A number of older versions of browsers including Chrome, Safari, and UC browser are incompatible with the new None attribute and may ignore or restrict the cookie.

This caveat meant that UA sniffing was the only reliable means of setting cookies that were guaranteed to work properly.

1: https://web.dev/articles/samesite-cookies-explained

This had nothing to do with feature detection, so the usual suggestions simply won’t work.

You don't get that sort of speed without heavy optimisation. In fact I'd be amazed if www.google.com isn't the most heavily optimised page on the internet. So it's not at all surprising to me that they optimise based on what browser is asking.

basically you can send slightly less data by only sending the relevant layout/js code etc. with the first request instead of sending some variation which then the browser selects/branches one

most likely a terrible optimization for most sides maybe not google search due to how much it's called

but then Google is (as far as I remember) also the company which has tried to push the removal of UA strings (by fixating all UA strings to the same string, or at least a small set of strings)... so quite ironic

UA sniffing should be a last resort but there have been times where browsers have claimed to support something but had hard to detect bugs and the cleanest way to handle it was to pretend the feature detection failed on those older browsers.

This is increasingly rare – the last time I had to do it was for Internet Explorer – but I would not take a bet that nobody will ever have a need for it again.

As an example, I recently had to deal with an exception to a TLS 1.2 requirement because while almost all of our traffic is from modern user agents, there was a group of blind users with braille displays using Android KitKat which defaults to only supporting a maximum of TLS 1.1 and “go buy an expensive new device” isn’t an appropriate response even though you might be entirely comfortable saying that random internet crawlers getting an error is okay.

You either support older TLS or you don't. Do not try to fine tune it to one known audience. You just locked everyone that is not in the sample you just whitelisted(!) out of your service. But you don't know about them, so no harm? If you think one use case for tls1.1 is OK for one use case, just accept you support it.

Same for every thing else. Every time you use UA sniff, you are doing it wrong. not matter how smart you think you are. You are not. It's just a momentary feel good because you are oblivious to everything else. Just like you felt good when you blocked tls1.1 before knowing of those blind users. Now you felt good allowing those blind users because you still don't know about another group that you might learn tomorrow or never, because you are blocking them :)

Just an evil company really.

It's fascinating how people clamor for more government regulation and control at the first sign of trouble, without reasoning about the implications of those things, despite a history replete with examples of power abuse from the same.

If the laws were still being enforced, a company that has a very large market share in web browsers, and a very large market share in search, would have to be extremely careful not to "accidentally" break their competitors' products.

Bro you have posted this on an article where GOOGLE fucked up PARSING A USER AGENT STRING to the point they deliver an empty HTML document. What feature on the Google search page even NEEDS such feature switching based on user agent? Google search worked pretty well on internet explorer 4 for heavens sake

But please, keep telling me how private sector is so much less incompetent than the public sector

However wanted to screw Firefox users the same way as the YouTube slowdown.

There's a silver lining though: it really doesn't matter if it's due to negligence or malicious sabotage, and if they (where "they" is hypothetical leadership within Google that is trying to eliminate Firefox) thought that pleading negligence instead of sabotage was going to be a good defense, we'll see how that holds up to future scrutiny and regulation. I mean hey, Microsoft tried to pull a lot of shit too, and they were probably more competent at it if the Epic Games lawsuit is any indicator of Google's competence right now.

Google is Firefox' primary source of income, and Firefox is strategically crucial token competition that stands between Google and an antitrust lawsuit of the sort Microsoft faced in the early 2000s (where Microsoft were basically a technicality away from being broken up).

If they wanted to sabotage Firefox I doubt they would choose to send a blank page. They would probably send infinite captchas or just disable most features (which they already do).

Sure search was their first product, but they have long since pivoted to being an ad company. And while yes, they can show ads along side search, the real cash cow is all the juicy data they can exclusively hoover up through Chrome to better target ads all across the web not just search.

304 points by hashhar 49 days ago | 122 comments

https://news.ycombinator.com/item?id=38349357

Middle management finds a tactic that implements the friction strategy - small "random" breaks and persistent performance issues.

An engineer would find the precise measure to implement - break the UA string sniffing targeting a specific browser.

Interesting theory, but that's exactly what M$ got dinged for in the antitrust suit re: Netscape.

If the risk is a lawsuit after years of anticompetitive behaviour - if they're unlucky - that's absolutely worth it for Google. Chrome is now THE web browser platform with the only two remaining notable exceptions being competitors in name only, and no single lawsuit can just reverse that.

Google is doing more "eternal irritation by 1000 papercuts". Never one big blow, never aiming to kill Firefox, they're financing it after all. Firefox needs to exist but never be attractive to users, especially on the phone where tracking is next level.

I think plausible deniability does a lot of the heavy lifting here. But statistically speaking Firefox being the one browser hit constantly by a stream of random Google issues can't be random.

The legal and regulatory landscape changed a lot since then too, with Big Tech slowly but constantly lobbying and pushing a lot more than just these tactics into normalcy. A lot of what's normal today was outrageous in the '90s.

Assuming competence and intention is foolish.

> An engineer would find the precise measure to implement - break the UA string sniffing targeting a specific browser.

You got the strategy right but the implementation is laughable, sorry :-)))

The implementation is: "We have a budget of N story points this sprint to resolve bugs, let's prioritize them. Let's prioritize by impacted audience size."

The audience size will make 99% of Firefox specific bugs be deprioritized out of the current sprint. And the next one. And the one after that.

And unless a senior engineer stands up to update the prioritization criteria, plausible deniability forever.

Repeated incompetence from one level is actual maliciousness from the level above, all the way to the top.

I suspect that these problems are more to do with neglecting testing, and just not caring very much about non-Chromium browsers

(Disclosure: I'm a long-time Firefox user)

A bug that breaks others' tools but not yours is not implemented intentionally, but it's not fixed intentionally, so it looks like a benign error.

At the end of the day, you use the chance you get, and it makes real damage while you sip your coffee...

Reverse search is now completely useless for example.

Not saying that they shouldn't have detected this (i.e.: where is the automated testing for those things?), but I don't think this is Google screwing up Firefox on purpose.

And if Google really wanted to screw up Firefox, they would probably do a better job than User-Agent sniffing. Firefox already has an internal list where it applies fixes for websites (e.g.: setting a custom User-Agent), and the bug report actually comes from it. You can see it by going to `about:compat` page in Firefox.

They didn't test on it at all. It's a browser with sub-5% market share.

There are folks in Google who personally test on firefox and believe in it ideologically as an important peace of the web ecosystem, but that's not a company policy and detecting Firefox bugs does not gate feature release for most projects.

Mozilla has cachet with nerds like us because of its history. Not because it's particularly better than alternatives or because it has enough of a user base to throw its weight around.

In contrast, they do test on Safari. They'd lose double-digit percentages of users if they didn't.

The ecosystem, unfortunately, generally has room for three top browsers. I'm not even sure Firefox is number four these days.

For obvious reasons, I don't think Google is devoting any resources to optimizing the experience of navigating to google.com for people who don't want any history generated on their use of google.com.

I agree with you that the other features listed are positives, but they don't appear to be positive enough to push use of Firefox psst Safari, Chrome, or Edge numbers.

Count me in for the destruction of the do-be-evil giant, but we're far from it not being absolutely dominant in the space.

I didn't even know what Kagi was.

It has one killer feature: you can block Pinterest from spamming your results.

The top 7 blocked domain are just pinterest.

Edit: I'm not saying tesla is building a street view clone. I can't imagine they have the bandwidth for that. But the cars recognize construction, speed cameras, police cars, red lights, red light cameras, stop signs and read speed limit signs, which all gets sent back up in real time. They might have to pay to augment their traffic data until enough people without teslas start using the app.

An offering of what?

Just because a few techy or aware people use Kagi (or another alternative), is still a drop in the bucket of overall search engine choice

Google search is still widely dominant, as much as we might not want it to be.

At the time netscape and microsoft were giving out free browsers while fighting for control of the profitable httpd server market, and blocking competitor browsers "for security" or something else was the play book of the day.

Is it just me, or is this absolutely insane? When Google ships a bug, it's suddenly the responsibility of browser vendors to "fix" it at the browser level?

It happens all the time, you probably just don't realize it. There is special code in Windows for supporting/un-breaking popular applications, same with Android and iOS.

IIUC "Interventions" are typically injected scripts or styles and "SmartBlock Fixes" are exceptions to tracker blocking.

Here's Raymond Chen sharing some of the worst examples from the 90s

https://ptgmedia.pearsoncmg.com/images/9780321440303/samplec...

Google has marginal incentive to not kill Firefox (antitrust), they have no incentive to make sure they provide a good experience on Firefox, let alone test with it.

The issue seems to be isolated to the search home page, and Google Search hasn't exactly been associated with "quality" in several years. Internal rot and disinterest gradually chip away at QA.

at least not for firefox mobile

they probably test for chrome (desktop+mobile), safari (desktop+mobile), edge(desktop) and maybe Firefox (desktop) but probably no other browser

Not explicitly anyway. Google's testing strategy is as it has always been: do some in-house burn-in testing, then launch to a small number of users and check for unexpected failures, then widen the launch window, then launch to 100%.

In this case, the user base in question is so miniscule that no bugs probably showed up at the 1%, 10%, and 50% launch gates.

I never actually looked at what this does.

https://www.google.com/search?q=what+is+my+user+agent+string

https://www.whatismyscreenresolution.org/

Probably the backend sent the initial headers and start of the response to the load balancer, then stared rendering the page and crashed. Presumably `` is hardcoded as it is always sent for HTML pages and maybe fills up one of the initial packets or something. The rest of the page may be rendered and return as separate chunks (or just one large content chunk).

"Over and over. Oops. Another accident. We'll fix it soon. We want the same things. We're on the same team. There were dozens of oopses. Hundreds maybe?"

https://www.zdnet.com/article/former-mozilla-exec-google-has...

"The page is blank. The page is blank the page is blank."

For example if you see that errors are being thrown on your site and nearly all of them are the new release of Chrome then you can quickly narrow down it being a compatibility issue with the new version. Similarly if someone is making broken requests to your site at a high rate it may help you figure out who it is and contact them.

It can even be useful in automated use cases. If Firefox 27 has a bug it is reasonable to add very specific checks to work around it gracefully. When possible feature detection is better, but some bugs can be very hard to test for.

The problem comes from:

1. People whitelisting user agents.

2. Adding wide sweeping checks, especially if you don't constantly re-evaluate them (For example Brave versions later than 32)

In this case it seems that it is hard to actually conclude that a user-agent check was bad here. Google could have been trying to serve something that would work better on Firefox (maybe it worked around a bug). Of course that code crashed and burned, but that doesn't mean that the check was a bad idea.

I agree that user-agents are often used for evil, but they can also be used for good. It is hard to justify completely removing them. With great power comes great responsibility. Unfortunately lots of people aren't responsible.

I'm still using Firefox because I'm ideologically motivated to, but it's no wonder that so many users are dropping the fox for chromium browsers.

https://bugzilla.mozilla.org/show_bug.cgi?id=1866409

If people don't want to pay for Kagi, then use Brave Search. DuckDuckGo was really gone downhill, so it's hard to recommend that.

I've seen this a lot lately, but no one has said why.

I solely use DDG and have done so for a long time. I have not noticed any specific changes, nor any degradation in my search results. I don't pay much attention to announcements or anything, so maybe I missed something?

Can you or someone please tell me how/why DDG is suddenly not recommended and "gone downhill"?

Switched to DDG ages ago, and even though it's not amazing, it's already better than Google.

Although, if we're honest, searching these days is absolutely atrocious everywhere. Any keyword search will only net you ads or "top X" articles full of SEO garbage most likely written by AI at this point...

I used to advocate for DDG, but they've gone so far downhill that I rarely waste my time with it. They've decided to go PG-rated everywhere even if you have safe search turned off. This is a problem because, even if they just want to block porn or illegal things, this can have an impact on legitimate academic research. And yeah, SEO trash floats to the surface, though they aren't entirely in control of this.

Kagi (which relies on Brave Search instead of Bing) has less flotsam to start with, provides lots of great tools to filter out the garbage, and is subscriber-driven instead of advertiser or investor driven.

DDG had potential, but they decided to give into moral panic and coast on their modest success instead of making substantial improvements. They also decided to pull a Mozilla and spend unnecessary effort working on a browser when they should be focusing on their core product. It'd be one thing to make a browser if their core product was actually good, but the best they are offering is a pinky-swear that your searches are private.

This is misleading. Kagi uses multiple external sources including Google and Brave as well as their own internal indexes

https://help.kagi.com/kagi/search-details/search-sources.htm...

DDG works well enough for me to be my daily driver, but it's absolute garbage for any kind of news search. Bing prioritizes MSN repost spam from dubious sources, so I'm better off going just directly to news sites I trust.

I think the only time I search for news (usually on Google) is to find a reliable article about a celebrity death.

In fact, you can identify the exact moment when the news about the death leaked, when the first editor adds it to the article. Generally, experienced editors will keep it out until a reliable source is provided, which typically happens quite quickly.

In fact I've found out about a number of celebrity deaths, just by having them on my watchlist.

Aside from the fact it's another HTTP request, but these days on the majority of computers and connections, that's a trivial thing.

I'll add that !aw to search the Arch Wiki and !aur to search the AUR are my two most favorite commands.

Because if I wanna search Reddit, I'd prefer to have '!r' search directly within Reddit and not litter my browser history with tons of duckduckgo entries like this:

https://duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%...

doesn't hurt to have options! to each his own

You’d think DDG could avoid this via JavaScript if they wanted to. Might be better for privacy, if not for their usage stats.

I have `s` to search sourcegraph.com, DDG has !sg. `t` to use Google Translate, DDG has !gt, `i` to search IMDB, DDG has !imd or !imdb, v to search animated images, DDG has !gif, `ni` to search nixpkgs issues which DDG doesn't have (although they do have one for the nix repo at !nr which is pretty impressive. I use `nc` for this). Not to mention personalized shortcuts like searching my company's GitHub organization or JIRA tickets that would never be a public bang, much less have a 2 character shortcut.

I stopped there but it is clear to me that I benefit from making my own short aliases for the searches that I use most. Plus it is nice to not send my logs to a third party and get improved performance.

Then you can "w chemistry" to open Wikipedia with a list of suggestions from the same Wikipedia ("Chemistry (Girls Aloud album)") in the same place

https://duckduckgo.com/bangs

I guess I could implement them as a web server, and point all my devices at it.

There is no point in making 100% of your experience worse just because you can't make 100% of your experience better

Also, as it is, I make 100% of my experience better by switching all the browsers to a search engine that provides better results than google and has !bang support.

There are at least two such search engines: duck duck go and kagi.

Wish they also allowed changing the !prefix

I also use Kagi lenses and it's been good, though for me the killer Kagi feature is being able to uprank/downrank/pin/block domains. Such an obvious and simple feature, such a powerful effect.

There is a psychological barrier to overcome in paying for searches, but once you get past that, Kagi makes a ton of sense. I suspect the reason Google never implemented personalized search results (like Kagi's where you can uprank/downrank domains) is because it's not about what you see, it's about what they show you. i.e. ads.

Yes! This alone is worth the subscription to me.

And I despise and avoid software subscriptions, Kagi is my third.

Also, YouTube is still the best video site despite going full mask-off on adblockers.

Then set up a gmail forwarding rule.

The full documentation is here:

https://www.fastmail.help/hc/en-us/articles/360058752414-Mig...

I honestly spent more time trying to wrangle the data in all of the other numerous Google services those accounts had data with.