（评论）

（评论）
(comments)

原始链接: https://news.ycombinator.com/item?id=41517272

航行者号航天器于 20 世纪 70 年代发射，由于借鉴了之前的水手号和先锋号任务的经验教训，进行了广泛的规划和设计，因此能够继续自主运行。尽管已有 50 多年的历史，Voyager 的系统仍然灵活且能够远程控制。尽管相关人员发生了更替，但拥有关键知识的关键人物仍在继续为该计划做出贡献。 Voyager 团队面临着独特的挑战，包括远程诊断问题、使用最少的工具控制硬件以及根据有限的信息做出决策。为了克服这些困难，他们采用了各种技术，例如使用模拟进行测试和故障排除。 Voyager 目前面临的一项挑战是处理老化组件和累积磨损。最近，一条被二氧化硅堵塞的燃料管道引发了一个问题，展示了时间对航天器材料的影响。然而，航天器仍然保持功能，并且仍然拥有足够的燃料来调整其轨道。尽管航海者号年代久远且位置偏远，但它仍然启发并提醒着人类的聪明才智和决心，提供对太阳系外围及更远地区的深入了解。记录和保存航行者号任务历史的努力仍在继续。

I continue to be fascinated by how they:

1. Are able to diagnose the problems remotely at such distances and on such old hardware. How can they even measure the thruster tube apertures here?

2. Decide what actions to take. It's not like they have a local test device to experiment on is it? (Even if they did I can't imagine how they'd reproduce the conditions of the real thing.) And if they choose poorly, I'd assume the mission's over. There's no replacing Voyager 1 if they brick it.

3. Have such fine control over the hardware. For something built in the 70's when RAM was largely measured in kB, they seem to have an insane amount of flexibility to remotely reconfigure the equipment. Whatever they did, there must have been some real foresight.

In short, the Voyager spacecraft were designed (after considerable design and operational experience from Mariner and Pioneer spacecraft) to be able to operate for a long time largely automatically without too much hand-holding. All major systems are multiply redundant and may be remotely turned on and off. While there has been personnel turnover on the program, it has not been of a magnitude to jeopardize program continuation. Finally, program management has been media savvy and well politically connected ensuring that operations are still funded. (Contrast with other missions such as Magellan to Venus which was deorbited while it still had propellant reserve leaving some portions of the planet unmapped)

JPL has (or had?) a "retiree badge" program that permitted retired staff to continue to access their office. Many programs benefited from highly knowledgeable personnel essentially continuing to report to the office every day without pay (not being paid comes with the luxury of not having to worry about being laid off if your charge accounts don’t have enough funds!) It was an absolute privilege to learn from these people.

I think it is inevitable that organizational "culture" changes. The tricky part is figuring out exactly what parts need change and what parts don’t.

For example JPL used to have beauty pageants ("Miss Guided Missile"). More recently, management appears to be trying to adopt policies and procedures from some venture funded commercial space companies. It is not clear how helpful these efforts will be given that these organizations are in fundamentally different businesses.

https://www.jpl.nasa.gov/images/slice-of-history-70th-annive...

Pageants were far and between and everyone knew participants had to prepare and they are not walking like fit/dressed that day to day.

Nowadays we have eternal pageant on the internet and lots faking it like they live/sleep/shower in full makeup and perfect fit.

I wonder what the fact of the matter is. It'd be rather hilarious if Humans have been shooting ourselves in the feet for the last few decades with many of our social engineering initiatives. And the reproduction rates chart is ugly.

I wonder what the fact of that matter is. Are reproduction rates dropping because women aren't as pressured to bear children due to those silly 'social engineering initiatives'?

Modern information access seems to have led to the sinking ship feeling, across many of the most prosperous countries, and so effectively that world wars, deadlier pandemics, and thermonuclear brinkmanship can scarcely compare.

Run off the mill MBAs are not allowed anywhere near to stuff as important as Voyager.

All the web apps, that run of the mill devs are implementing are nowhere near as important as voyager.

I think we are fine ;)

I guarantee you that a run of the mill MBA program will say something about institutional knowledge being important. You’re needlessly and unjustifiably finding an excuse to dunk on something that you know people here dislike, despite it having very little to do with the actual topic. Which…fine. But you’re also so keen to do it that you aren’t even correct. Please stop projecting every gripe you have with your boss onto this mythical “evil MBA”.

No, you need to think like an MBA.

It means that they are the most valuable employees, because their productivity per dollar is infinite.

It also means that when you sort all the (non-executive, of course!) employees by total comp in preparation for layoffs to make the budget look better, they are at the bottom of the list.

Yeah! Being almost 50 years old it's not like people is not working there anymore but that probably a bunch of people in the original project has already died!

Great forward thinking

We do something similar at my job for the greybeards that were very influential on our current projects. They "retire" but are retained as a very part time employee that only get paid if we need to bring them in for a day or two to help answer some questions. The managers love it because they don't need to find work for this person, pay for any benefits, or need to get approvals to pay them like contractors. As long as we keep doing relevant work to their expertise, they will continue being retained. There is a limit for how many hours they can put in because these people are incredibly expensive as they usually retire at the top engineer level and retain the equivalent hourly wage for that position.

Paying someone as sporadically as you would a contractor but they’re only getting their usual hourly wage? Doesn’t sound expensive to me. Sounds like a ridiculously good deal for the employer.

What blows my mind is the organizational knowledge needed after so many years to keep it going. You don't just hand a guy some man pages when he comes onto the project. I'm sure people have aged out, yet they still understand the complexities in the design. That is something we need to understand and prioritize in the systems we build today.

When Voyager failed last year with a CMOS memory error, one of the big problems was that a bunch of low level information was gone or conflicting. For example, they sometimes had to guess assembler instructions because the code printouts were low quality photocopied pages. Or because there were handwritten comments or comments scratched out with pencil without any clue about why it was done.

One saving grace was the fact that the architecture and the code space was simple enough so that they could reason through the symptoms and actions to take, something that would have been much harder with modern spacecraft.

Check out this amazing talk: https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__

Startup: "Do you remember when we inserted this quirky module running in AWS? We can use that to implement this next feature. That was a useful decision!"

Voyager: "Do you remember when our parents inserted this quirky module that has since left the solar system? We can use that to turn it off and on again. That was a mission critical decision!"

Not parents for most of us. Grand or great-grandparents. The senior engineers were highly educated and experienced, implying ages 30-50s during development. They are in their 80-100s now.

> It's not like they have a local test device to experiment on is it?

I would bet they have one or more simulators (“digital twin” in the parlance of our times). I’d want one simulator to always reflect the current state of the probe (with state data assimilated periodically from the real probe). Then other simulators can be used to test management changes to see how the system might respond.

I would imagine if the design/assembly information was broadly available (internally) in the past, there's probably one or several "digital twin" emulations of the craft, or at the very least specific subsystems of it's computing resources. There must be some kind of analog/simulation of it's software just for proving "bugfixes" before upload, like the coms error and subsequent setting of the "solar system record" for "furthest distance remote code update" earlier this year.

There isn’t a simulator or digital twin for voyager. It has a bespoke processor made with 74* style logic. One guy will puts together a command and they will have a review where the other engineers will try and independently verify it. Then they copy and paste the command somewhere to “run it”. It happened, fairly recently, that the command had a typo that was caught in review, but the “wrong” pre-review command was used and the attitude became off by so much that they lost contact. It was only by cranking up the power at Goldstone that they got a command through. This fundamentally changed their understanding of the largest angle for which they could still communicate with the spacecraft. They just hadn’t wanted to try larger angles before because it was too risky.

2 weeks ago, Bruce Wagoner from the Voyager program gave a talk at !!Con about how they recovered from the CMOS memory issue that they had a year ago.

It’s basically blind debugging with a latency of 45 hours.

The talk is amazing and goes through the computer architecture of the spacecraft as well as the challenges of dealing with something that is so old, with so documentation that has conflicting information or unreadable etc.

https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__

- "A fuel tube inside the thrusters has filled up with silicon dioxide, a side effect of age within the spacecraft’s fuel tank."

Where the heck do you get SiO2 from on a spacecraft? Some kind of silicone?

edit: "clogged with silicon dioxide, a byproduct that appears with age from a rubber diaphragm in the spacecraft’s fuel tank"[0] —I'm guessing that is a silicone rubber. I didn't know that rubber can decompose into sand.

[0] https://science.nasa.gov/missions/voyager-program/voyager-1/...

The Voyager mission is such a wild achievement. Both the sublime design and craftsmanship that must have gone into the hardware, and the deep institutional knowledge required to keep it running is awe-inspiring.

It's been operating for 47 years and it still has fuel left to make attitude corrections. I wonder how they managed that feat.

Not to downplay how impressive the Voyager probes are, but it seems they packed a fair bit of hydrazine. From "Engineering the Voyager Uranus mission":

While it was not a design requirement, the option for an extended mission past Saturn was always protected, unless it meant compromising a major mission objective at Jupiter or Saturn.

Even though the probability of Voyager 2 lasting another five years was calculated to be in the range of 60 to 70 percent -- well below NASA's usual acceptable probability-of-success threshold -- the decision was made to send Voyager on to Uranus.

After its Uranus encounter, Voyager 2 still carried 48% of the hydrazine initially loaded into its tanks, eight-and-a-half years before.

[1]: doi:10.1016/0094-5765(87)90096-8 (can be found on the hub of science)

No, way too little sun out here. The sun is just a dot far away. The Voyagers both use RTGs, radioisotope thermoelectric generators. Decaying plutonium, essentially. That's for the electrically powered systems. The thrusters use liquid hydrazine, which is common for those kind of thrusters.

Edit: There's more about that in the NASA link in a sibling comment.

Solar has nothing to do with it. Voyager uses hydrazine, of which over 80% has been used up over the years. They simply use not that much of it as it's not for thrust, but for aiming at Earth more or less.

I've heard good things about some Voyager documentaries, and I've wanted to watch one with my daughter, but NASA keeps making the documentaries out-of-date and incomplete. How many amazing stories will there be to tell by the time Voyager is truly beyond our knowledge?

You mention voyager documentaries but fail to mention which one. There's lots, and its impossible to know which one is good until you finish it. Please tell us which one you wanted to watch.

It's sad how much documentation is honestly unreachable now.

（评论） (comments)

（评论）
(comments)