It is basically a terminal server, proxy, bastion or anything similar. You log in there with federated identity (for example AD) and it logs you in into target system with some shared or temporary user. Usually it also records session and does other security/compliance related things.

Examples are Delinea (former Thycotic), CyberArk, To some extent Apache Guacamole can be used as PAM.

Everything should be 'hardened'. This also means everything has to do proper authentication and authorisation, and skipping that step by letting some proxy do that just creates a bottleneck in security, reliability, availability, and performance.

It also doesn't really matter how it's done, a Kerberos Ticket, x509 client certificate, JWT or multiple credentials (i.e. username with a password and MFA token) are all plenty valid. Granted, a ticket, certificate or token allows shipping claims or attributes allows for directory-less access control, but that doesn't mean that having to do directory lookups is not feasible anymore.

Most of the other things like f5, pa, Citrix, powerbroker and bomgar are just really shoddy software that you setup to attempt to not have to bear responsibility or know what you're doing (or it's clipboard/checklist-based security...), but that just bypasses fulfilling the actual need of a good IAM and PAM implementation. None of those products do it better than what is natively supported, and they are consistently more problematic (be it performance, cost or actual security).

Thycotic had some vulnerability with a symmetric recovery key a few years ago. But comprehensive product like this or roll your own this is frequent so I'd rather do keys and certs like others suppose

a) relatively complex to set up properly if its not a Windows in AD

b) an SSO, not federated ID. Which means reasking MFA is not possible.

https://www.freeipa.org/

https://www.freeipa.org/page/V4/OTP

Isn't reasking MFA on purpose just a way to make people hate MFA? Shouldn't everything support a "Trust this device" option and then never ask again from it?

Managing access to privileged systems at scale is hard despite these tools. This isn’t a knock on the tools. It’s a people/risk problem. Eg staying on top of those recorded sessions becomes increasingly difficult over time.

The sustainable way is to work with business and compliance and remove the need for privileged access. It’s doable in many scenarios and leaves a very small “rump” of things that do need privileged access, with all the attendant overhead.

In general, challenging teams to do without privileged access and making it a “last resort” thing is a great idea, depending upon your industry / risk profile.

Used Powerbroker and cyberark for a long time and while they're good at stated purpose the integration with more flexible and modern auth systems has had a lot of friction.

The particular regulatory area I work in is also just a non-starter for federated AAA from outside the regulated systems which colors my opinion though.

Combined with command restrictions in openssh and sudo etc you end up with several wholly disjoint attack surfaces, decent logging, and granular user restrictions.

Would you care to share how you achieve this/what does the implementation of these two look like?

Captive portals are web auth pages for use cases the more structured method doesn't work for. They were envisioned as making you sign in hotel wifi and such but work in the other direction as well by forcing a web user login before allowing traffic from a host for some period of time.

(But I much prefer keys/certs over passwords where possible.)

> First, some vendors make it difficult to associate an SSH key with a user. Then, many vendors do not support certificate-based authentication, making it difficult to scale. Finally, interactions between public-key authentication and finer-grained authorization methods like TACACS+ and Radius are still uncharted territory

Keys (with/without certs) are the best route, but not always possible for every situation.

Another possibility would be to use CA certificates for authentication and only TACACS+ for authorization and accounting. Juniper now supports CA certificates. Cisco may in 10 years.

"All implementations MUST support this method"

https://github.com/fire1ce/UDM-Persistent-SSH-Keys

sshpass didn't work, ended up rewriting the whole thing in Paramiko ... only to find out it doesn't respect the http_proxy environment variable.

That was not a good day

    script -qfec "mycommand" /dev/null

I had enabled the debug option on expect and I couldn't see the password prompt when the program ran under cron (i was redirecting the script output to a log file). It did appear when running on the prompt though.

I couldn't figure how the sftp program was determining that it was running under cron. I suspect that it was inspecting if stdin was connected to a terminal or not, but I gave up around 4 am.

If you would run it in the background "./program &" and then exited ssh it would also exit. It wouldn't run with cron until: export TERM=vt100 script -c "screen /foo/bar/program" /dev/null

All the program did on the console was print to stdout. How are you even program shit like this!?

Just wondering... have you tried running it with nohup and in the background?

Avoid passwords and use keys instead (easier to distribute, easier to generate, lets you lock them down to a single command).

The above avoids much unnecessary thinking.

If you really have to, there's always sshpass. And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary, first try harder do it the easy way.

I should have explored rsync better. I asked chatgpt some use cases and it made seem to be a bad fit because I needed to also delete some files on the destination machine. My prompt fu was probably a bit bad at the time.

> Avoid passwords and use keys instead .

The vendor only provides password authentication in this case

> If you really have to, there's always sshpass.

I´ve tried it, but sftp, when running under a cron script, detects that it is not running in interactive mode and does not issue the password prompt. The problem might have been caused by the TERM environment variable not being set as another reader suggested.

> And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary

I've used -o RequestTTY=force, but it also didn't work. Granted, it was close to 4 am and I might have missed a key aspect.

> first try harder do it the easy way.

Paramiko endep up being easier once I understood how to use the proxy command to interact with the company's http proxy.

Then again, don't do this. Use rsync. It is more robust and will avoid other problems in the future. And don't accept that a vendor only supports password-interactive authentication, it is unlikely to be the case, nobody implements their own ssh and every standard implementation of ssh accepts more ways of authentication.

Paramiko is a perfectly workable solution, but it's way more complex and requires more maintenance for your eventual successor. Please don't be that guy.

It's not my orgunit that manages the contract with the vendor. I will absolutely use paramiko instead of trying to explain a technical problem to two middle managers in another orgunit to take it up with a non technical person on the vendor side whenever they feel like it when my deadline ends at the end of the year

Workarounds are fine as long as what you’re working around is documented and the business is aware.

Without at least trying to raise the issue, you are being “that guy”.

Even if avoiding good practices with PKI was defensible (and it definitely is not), further avoiding `sshpass` in favor of this more contorted trickery is (imho) probably the wrong choice.

It's an extremely common attitude unfortunately - you can see it all over the place, especially in the Unix/Linux world e.g.

* People thinking sysvinit (janky Bash scripts) are fine.

* 50% of Linux software doesn't work if you have spaces in your path. GNU Make explicitly doesn't support that.

* Over use of text based APIs, e.g. /proc and /sys.

> Ignoring failures because they're moderately unlikely is the hallmark of a bad developer.

Totally agree.

> runit

First I've heard of that. Looks interesting, but it doesn't seem like it has nearly enough features to run a modern desktop system? It just starts daemons and keeps them running as far as I can see.

Runit is pretty good - it's the default in the Void Linux OS that I use on my personal machines. Definitely good enough for a modern desktop system, though it's not aimed at the same crowd as Ubuntu and Fedora.

Well, a) that's fine for interactive use but awful for unattended use because it's so likely to go wrong, and b) it is actually pretty easy to beat that - JSON and jq is much easier, nicer and more reliable. Or Nushell/Powershell structured pipes. Or an autogenerated properly typed Python interface. Take a look at /proc/self/status before you tell me you'd rather use some awk/cut monstrosity than `jq .ppid`.

> If Fuchsia had beaten out UNIX in some parallel universe, it does look like we'd all be happier.

I'd definitely be happier - I wouldn't spend so much time debugging why Linux stuff breaks!

2) No idea where you got that from, I never had a problem with this (as a full-time Linux user)

3) Mixed opinions on this one. There's a ton of various info in /proc that could theoretically be exposed via different syscalls, or maybe a single syscall? But having a text-based API in this case isn't a big issue really.

2. Try putting a space in your home directory and let me know how that goes...

3. Most programs don't resort to reading /proc or /sys because it is such a pain! I bet there's a ton of undiscovered vulnerabilities in programs that do.

2. Another fallacy. Obviously there will be badly written programs that don't handle paths properly. And I'm sure that if I put a space there it will screw things up. But it doesn't mean that most programs do that?

3. It's not a pain. It's absolutely trivial to do, and people whose code is vulnerable in this case are most likely just bad at programming and shouldn't be writing at such a low level as to cause vulnerabilities anyway. These are the type of people referred to as "developers" instead of "programmers". (Source: I personally parsed /proc entries without third party libraries, can't say it was hard)

Who wants to have to be an expert in shell scripts to have to write them? In any case you're still wrong. Experts aren't immune to footguns. Post a complex shell script you've written. Let's see.

> Obviously there will be badly written programs that don't handle paths properly.

Like GNU Make?

> It's absolutely trivial to do

Trivial to do so that it works for you. Absolutely not trivial to do so that it always works.

The types of people who know the difference between those are referred to as "senior" instead of "junior". (Source: I personally parsed /proc entries too without third party libraries and there were plenty of footguns that a junior developer would skip over.)

If you aren't good with shell scripts, why bother? Go use Python or something else entirely. And it's not like you would use shell scripts to write all your programs, they aren't suitable for anything more complex than stiching a bunch of programs together anyway, yet people still try. And then those people complain that shell script is full of "footguns" when they are misusing it.

> Trivial to do so that it works for you. Absolutely not trivial to do so that it always works.

No idea what you are on about, literally every single detail you need to correctly parse /proc entries is in proc manpage. And yes, if you wrote your parser correctly it will still work between kernel versions, because /proc is considered a stable API between the kernel and userspace (same as syscalls). There are no "footguns" and no mysteries to it. It's a solved problem. Provide a concrete example of a footgun if there is one, and by "footgun" I mean something that isn't clearly stated in the manual that everyone dealing with /proc reads (right?)

> If you aren't good with shell scripts, why bother? Go use Python or something else entirely

I agree! And given that essentially nobody is "good with shell scripts" we can simplify that advice to "don't write shell scripts".

That's some kind of logic right there. If people don't read the manual, they have every right to expect that things will break. It's not surprising or shocking, it's just user error.

> we can simplify that advice to "don't write shell scripts".

No, we cannot simplify it like that. Shell scripts are a brilliant tool for a certain kind of problem, and they work well when used for that class of problem. When used for anything else, they work poorly. Same applies for literally any other tool. It's like saying "essentially nobody is good with a CNC machine so don't use it"

Workarounds are many for network devices it seems!

What it does do well:

1. can validate a key signature issue

2. firewall port-knocking (extra http ports interleaved with instant ban ports)

3. ssh over https setup

4. IDS tripwire Morse-code knocking

5. reverse-proxy configuration for trusted zone ingress

This approach helps solve several issues:

i. distributed firewall probes or nuisance traffic

ii. brute force attempts or nuisance traffic

iii. obscures security posture identification (what got an IP blacklisted might have occurred several minutes ago)

Indeed, I also

Good luck =)

Can anyone enlighten me why sshpass is broken, or explain the examples on that page?

On the sidebar, in the about, it says,

> sshpass is broken by design

This is another repo. But probably where they got it.

More importantly though is that sshpass keeps your password permanently on your computer, thus increasing security risks considerably.

I can argue that SSH password auth only makes sense.as in interactive affair; for non-interactive auth cases, there are public keys, certificates, smart cards, etc.

Do you keep all your passwords in one directory or in something more complicated?

  User: user
  Pass: 1

I don't know if that is efficient for 30K machines though.

I was also solving more problems than just ansible/puppet solve... like monitoring of specific systems on the machines.

Regardless, more than one way to skin a cat.

You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers.

There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.

Again, many ways to skin the cat, but at the end of the day, this solution really worked extremely well. I would do it again in a heartbeat.