If you're a normal HN reader that found themselves interested in this story, go check your password reset feature, specifically the email association logic!

Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid.

Based on the other comment that describes how this bug works, it is completely trivial to avoid. If you use a statically typed language, you'd have to go out of your way to create this bug, and it'd stand out like a sore thumb in code review, to the extent that if I saw that I might wonder whether my coworker is actively trying to create a backdoor.

Something like Java is going to have the largest hiring pool, the largest ecosystem, and support from every major vendor. I don't particularly like it and think that Scala is pretty much a straight upgrade, but Java's a smart choice.

My general take is: glass houses, etc. And: everybody is in a glass house.

Maybe Spring has had issues. Like I said I don't actually like Java and all the annotation stuff. I avoid it, but it's still going to be one of the more robust choices you can make since every major enterprise uses it. Java had log4shell, which was pretty bad, and probably wouldn't have happened if Java had had string macros. It's one of the reasons why Scala is a better language: people use things like compile time macros which prevents those sorts of mistakes. It looks like akka-http for example has only had DoS vulnerabilities (e.g. getting zip bombed) + some niche stuff like request smuggling if you used it to build a reverse proxy or usage of directory listing on Windows.

That said, sure, large frameworks all have bugs. But not being able to tell whether you are dealing with a string or an array in your business logic is just silly.

Same with something like SQL injection or log4shell. These are also preventable using string formatting macros. If `DatabaseConnection.run` takes a `SqlQuery`, not a `String`, and you can only make a `SqlQuery` through a safe API, and your escape hatch is a macro that does query parameterization instead of string substitution, then your users can't get it wrong. This is how Scala libraries actually work and have worked for years. Similarly if you build your logging library to take MDC objects that are built with format macros. These are solved problems.

Today, it has been superseded by FromSql which uses string interpolation API so that it is transformed into safe parametrized queries without any explicit action from the user.

None of that is a reason to avoid C#! I'm not trying to say that using the two most fertile sources of web application security vulnerabilities over the last 20 years is per se a bad decision!

That is because that is what enterprise uses, I don't think it would have changed a lot now.

When Facebook was founded there weren't actually many good alternatives to PHP.

On top of that it looks like they had a 9.6 CVE that allowed integrations to perform commands as other users...

From the outside it looks like they are trying to ship features faster than they can keep them safe and tested. Perhaps because they are having an incredibly difficult time monetizing well? It makes sense from business standpoint in some respects, but also the security stuff could just absolutely tank the business when the whole point of a (self)hosted git solution is essentially just account management.

One of which was around credentials resetting to emails that aren't stored in the API auth system itself, but rather come into Salesforce as a support case. "Don't worry, a support team member has to action the request" was meant to be reassuring, until I explained that this translated to "the only mechanism in place to prevent credentials being stolen comes with a massive social engineering vulnerability".

But it's the previous choices I haven't come across yet that worry me.

this sums up their entire product

every feature you could possibly imagine, somewhat working

They even went overboard with the transparency and made public some slack conversations which for me would have made it one of the worst places to work.

That's an unfair comment. Even the best teams ship bugs. If you want to measure the quality of a security team, you look at their performance trajectory (for both detection and response) relative to the size of their total threat surface.

My favorites are

* using included files that run no job is a failure. The only real work around is adding a noop job all over your ci system.

* try to use code reviewers based on groups. The logic is so complex and full of errors i can’t even explain it unless i spend an hour reading the docs.

* when using the merge train and enabling merge result pipelines you end up with two different jobs per commit. This is cool except in the UI it always shows merge results first. If you have ten commits you need to look on the second page to find the most recent commits ci jobs. That is just annoying but more no environment variables overlap for what MR or commit it is. This makes doing trivial things like implementing break glass pipless almost impossible.

Anyway gitlab sucks i wanted to not use github but really it’s just bad. Not to mention we have outages monthly that we always know of 30 minutes to an hour before gitlab does then we look on the status page and see the downtime is 10 minutes when its been 40 for us and likely everyone else. We have in the last year had close to 2 full days combined of downtime from gitlab. Of course they report 99.95% uptime.

there were some "addons" like panda something that made it less worse, but still a crap fest in terms of usability and compliance.

not to mention that now you can barely use it without being logged in. im overall glad to have moved to gitlab and codeberg. do not miss github AT ALL.

I just had an idea, maybe using a + alias ([email protected], made famous by gmail) could help against attackers. Even if they find out your email they will never guess the part after the plus. If you forget it though then you can't reset your password anymore either.

If you struggle with memorizing your username/email, there's a near zero chance you're using a password manager, which also means there's a near zero chance you're using decent passwords for your logins, in my experience.

So nothing ever is 100%.

Your service using emails for logins or adspam or whatever now faces a choice. You probably have to accept periods, and you probably don't want to try to hard-code all the different ways a period might be used legitimately as opposed to a typo, so you have to deal with that problem somehow. You can canonicalize (opening yourself up to hijacks, some unintentional as legitimate users just have emails that clash in your system), or not (potentially locking out some users).

All they have to do is set up a SMTP server and wait for junk mails, thereby learning about the e-mail addresses. Say Walmart sends some flyer. Poof, they have that user's e-mail, and the fact they are registered with Walmart.

    The vulnerability lies in the management of emails when resetting passwords. An attacker can provide 2 emails and the reset code will be sent to both. It is therefore possible to provide the e-mail address of the target account as well as that of the attacker, and to reset the administrator password.
    
    Here's an example payload:
    
    user[email][][email protected]&user[email][][email protected]

    emails = request.POST.getlist("email")

But then using the email address from the request rather than the already verified address in the DB seems like a weird design decision to me anyway.

I mostly hate the way strong params gets used - it's a bad compromise between letting Ruby people do Ruby things and trying to plug up a category of vulnerability that's been biting rails apps for a decade. Now I do all my api definitions in openapi and it's way easier. I haven't tried it with a rails app but I think it'd work well there.

If I had to guess what they did was:

  user = User.find_by(email: params[:emails])
  params[:emails].each { |email| send_recovery_email(user, email) }

  user = User.find_by(email: params[:emails])
  send_recovery_email(user, user.email) if user

Edit: Someone has digged it out: https://news.ycombinator.com/item?id=39162126

The bug is accepting an array when it should only take a scalar.

The design error is that the endpoint should not be taking email addresses at all. It should take account IDs.

Even if a system uses email addresses as account IDs they are conceptually not the same and the code should not muddle them.

Keep them separate and then even if you get an "allows an array where it should have been a scalar" bug the result should be either just the first account in the array gets a reset email or all the accounts in the array that are existing accounts get reset emails for their accounts.

I'm just saying that in the software and in the database store account ID and email separately. Treat the fact that the account ID column matches the email address column as just a coincidence that you do not take advantage of.

I'd enforce not taking advantage of it by having employee accounts actually use an account ID that does not match their email address, such as their name, so that if we accidentally leave out a call to EmailFromAccountID(...) somewhere and try to use an account ID directly as an email address it will break employee accounts.

Also, it is not clear to me that even with user visible account ID that is not the same as email address that it would take two email rounds trips.

The reset page could take email address, not account ID. The reset endpoint could then look up the account ID from the email address, and initiate the reset, calling the SendEmailToAccount service with the account ID to send the email. That service would look up the email address for the account.

I disagree. It's a bonehead mistake to send password resets out to tainted email addresses. As this was an authentication change it should have received extra scrutiny and so have been even harder to introduce.

Something is wrong with their engineering culture that needs correcting.

That's a statement.

Such as that time they deleted the prod database.

> Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid.

You should examine the history of security issues on GitLab. There are critical exploits multiple times a year, requiring an urgent upgrade of your GitLab distribution. Gitlab is the worst product I've used security wise.

Gitlab is both open source and has an on-prem product, so my guess is that you're simply hearing about more of the Gitlab bugs than you would with a comparably sized competitor.

It seems you might not be using their on-premises product, considering your guesswork. We used it for years and it was a nightmare. Almost every upgrade was problematic, and we often had to scour through GitLab issues to find solutions from other users. These solutions were often makeshift and carried the risk of causing further issues. Their salaries are below market rate, which reflects in the quality of staff they hire (there are few exceptions). I prefer not to point fingers, so I won't link to any specific discussions from GitLab. It's worth noting that they have a culture of open discussion, and from what I've observed, the engineering quality in some of the teams was quite low. We utilize numerous other large scale open source projects in our stack and have never encountered as many problems as we did with GitLab.

"so my guess is that"

> My point is that you don't hear about most vulnerabilities in SAAS products, because there is no norm of disclosing them. BUt disclosure is unavoidable for open source on-prem products.

I already addressed that point, explaining that we use other on-premises open source products of similar size, and GitLab was the poorest in terms of quality. I haven't drawn any comparisons between GitLab's on-premises and SaaS products, so I'm puzzled as to why you continue to 'guess' the reasons behind our experiences, especially when those guesses have been evidently incorrect.

Once a year they do a major release, usually around May, and I need to upgrade Postgres or Redis but that's the extent of it.

Every product has security issues and what should worry you more, things that never see security patches or something that does?

https://gitlab.com/gitlab-org/gitlab/-/commit/c571840ba2f0e9...

    - recoverable.send_reset_password_instructions(to: email) if recoverable&.persisted?
    + recoverable.send_reset_password_instructions if recoverable&.persisted?

    # Concern that overrides the Devise methods
    # to send reset password instructions to any verified user email
    module RecoverableByAnyEmail

Anyway, in the fixed version it's still called RecoverableByAnyEmail. Do people not read the code around what they are changing??

Added 8 months ago [1]. And then one month later:

> "password_reset_any_verified_email"

Was removed. 7 months ago [2], *note* __verified__ word here.

No blaming or conspiracy intended in this post, just listing links to relevant commits.

1 - https://gitlab.com/gitlab-org/gitlab/-/commit/94069d38c9cd63...

2 - https://gitlab.com/gitlab-org/gitlab/-/commit/a935d28f3decf8...

Initially a single email could be passed into the API/form call and they would look it up. If found they would send a recovery to that email but it was the email the user supplied not what was in the DB.

Oh, no problem we looked it up so they are the same!

But then the ability to look up accounts from a list of emails was added. If any email matches the account lookup would succeed. Then they sent the reset link to that same user supplied value but OH NOEHS IT'S AN ARRAY NOW AND SOME MIGHT NOT HAVE MATCHED ACCOUNT EMAILS!

So they ended up sending out reset links to a tainted list of emails.

Rails "concerns" are the worst IMHO anyway, but looks like they aren't using strong params here either which is even worse. Also someone thought it was more elegant to reuse the tainted value which is par for the RoR course.

Basically a requirement for this attack is to know the email of the user you want to reset, but, there is a hidden email address that is tied to your gitlab userid (a number incrementing from 1).

Since its a safe bet that ID 1 or 2 is an admin: thats a good target.

the email is something like [email protected]..

Really bad, seemed like it was automated.

2FA saved us here.

And the worst part: On most services you can't even disable it, the only way around is often only Enterprise SSO.

On some services you can set up a phone number for SMS token instead. But I've never seen the possibility to require both. Password reset only with e-mail AND SMS token.

- Easy to accidentally forward confidential tokens

- Validation of email sender authenticity is still piecemeal, and there are relatively frequently ways to bypass or work around validation.

- Mail is not end-to-end encrypted. I hope that it's at least encrypted with TLS, but last time I actually messed with talking directly to an MX, it seemed like TLS was still limited to smart relays, and actual mail delivery always went to port 25 as usual...

- The cardinality of e-mail addresses is not really defined anywhere. Gmail has plus addressing and ignores periods for the purposes of delivering an e-mail. Meanwhile, Google Workspace defaults to not ignoring the periods. Pretty sure some mailbox providers are case-sensitive and others are not. This means you can't know if an e-mail address at a given provider is unique. Best bet is to treat the entire damn address as a bag of bytes, but that opens up room for UX issues of all sorts, so it's hard to balance.

- It's bad that if your e-mail address gets compromised, any account you have that doesn't have some kind of secure 2FA is literally seconds away from being compromised, too. Obviously, this is not a limitation that is limited to e-mail addresses, people wrongly treat SMS as secure, too. The thing with e-mail addresses though is that they're easier to treat as secure because it is somehow still less of a crapshoot than cell carrier security, and also because literally the majority of online accounts can be stolen with just access to the user's email address, and it can be done quickly and easily, and in many cases it can be hard to convince support that you are the correct owner afterwards.

And second because hijacking someone's email account opens up a lot of different services to the attacker. 2FA over imap is still not a thing with most services/clients. Some people log into their webmail with username&/password on untrusted devices, ...

It was some junky web interface to a spam appliance of all things, I'm not sure if it was intentional, or just some php rookie wrote the code.

One of our users discovered it when they had a (rare at the time) special character in their password.

I work for a huge government owned telco and our networking guys are the best. They keep us server guys in line. So even though they did expose our Gitlab to an extent, for certain external projects and consultants, you still can't visit it from the internet freely.

And also we manage users in AD so there is no SMTP connection to even do password resets.

But we really need to enforce more 2FA, we've left it up to each project to enforce their own rules on 2FA.

For a highly secure environment, I agree to practice even more defensible tactics, but I think software out to be designed to survive the open web.