I have seen this approach with C-python and NodeJS already and I think it simply not viable, what they are suggesting is compiling the runtime (the same one you use in non-wasm projects) to wasm and then run your python code on top of it.

This is a double-whammy of performance degradation, you basically have two JIT-compilation steps happening (once by the wasm runtime to compile the rust-python wasm and again by the rust-python code compiling your python code). And this is on top of normal performance degradation from using a dynamically typed language compared to a statically typed language

To make dynamic languages (even JS) viable to run in a wasm runtime, the language must be compiled directly to wasm.

Project still looks pretty cool and useful though, there is plenty of python code that could be useful to use in the browser no matter how badly it runs. Just don't try to build a web framework on top of this kind of approach.

Edit: Let me reframe this a bit, this is what I think, I haven't really benchmarked anything and rust python might be doing some tricks I am not aware of.

So really most business logic can easily be satisfied by "compile the interpreter to wasm and then run the dynamic language on that", and doing it this way can move existing "learned the hard way special cases" byzantine business code to something that can run on a web server and be accessed by the companies employees rather than passing around scripts for them to run, with a lot of benefits including instant upgrades for everyone for bug fixes.

That said, this specific impl claims to only support half of the standard library so I kinda doubt its ready for any 'serious' business usecases yet anyway.

Once it does start then your arguments can make sense, but even so it would still make it impractical for most things.

Trust me, when the Javascript dev tells you something will be slow, it WILL be very slow

My https://lite.datasette.io application usually starts up in less than 10s - most of that is downloading about 10MB of WASM blobs, and 10MB isn't actually that big these days (many sites serve more than that in image headers).

When I built Datasette Lite I did it as a research project, assuming it would be far too slow loading to be useful. I've since changed my mind on that.

10 seconds is absurdly slow, though. That's like time to install Mathematica from a disk image level slow.

I hate to say this, but have you used any $ModenWebApp with $HotJSFramework recently? I thank the gods when those pages load without a 5-10 second of fancy spinning animation. Really thought we would be in a better place by 2024 but nope.

A very rough estimate just now on my iPhone over 5G: 12s for the first load, then when I visited again took around 3s.

Indeed, the web demo takes about 5 seconds to cold-start on my beefy PC, between downloading the 22MB WASM blob and compiling it. It also grows the WASM heap to 160MB after running the simple Fibonacci example, and WASM heaps can't (yet) be shrunk, so the only way to reclaim any of that memory is to discard the whole instance and start over.

It's cool that it works, but not very practical.

You could just run RustPython as a native binary, or use ol' reliable CPython.

Granted, the environmental cost to all that extra energy consumption may not be palpable. Then again, you're exactly choosing Python in the first place if you care about the environment.

Only seconds? Today it takes 45 minutes to do the update by hand, but there's this handy 20-line Python script that does it instantly.

(Hypothetical example only, but it's not uncommon.)

Which means this whole thing is pointless from an end user point of view. The technology stack is getting very deep - Python, Rust interpreter, WASM, in a browser. I'd love to get back to running things on a toaster with no dependencies.

Or… you could just use Django. The framework built for running python on a web server.

The usual way to do this is get a Python interpreter, sandbox the hell out of it on your server, and then run the untrusted code but this obviates the need for security paranoia quite a bit since it's running in the user's environment.

If you go full RustPython, surely looking to eek out performance must be a major reason to even go there rather than just script with CPython like it was 1999?

In my experience this is very rarely actually true, even though people think it is.

You already counted that, that's the Python interpreter (in this case RustPython) overhead.

---

The steps are (1) RustPython interpreter (2) Rust compiled to WASM (3) WASM runtime.

"Normally" you'd have (1) CPython interpreter (2) C compiled to machine code.

I'm not sure how much overhead WASM runtime really is...I'm curious, but I doubt this is awful compared to CPython.

---

FYI, for CPython as WASM, https://pyodide.org/.

That's wonderful.

Next up, wasm supercompiler...

I'm not sure what you mean by that. The runtime doesn't compile WASM, it simply executes it.

There are tools for dealing with interpreter runtime overhead this by pre-initalizing the environment like Wizer[0]. ComponentizeJS[1] uses it to pre-initialize the Spidermoney engine it packages to gain fast startup times (and you can then prune the initialization only code with wasm-opt). As techniques like ComponentizeJS are also being applied for a specific set of interpreted files, you can even prune parts of the interpreter that would never be used for that specific program. If you want to go even further you could record specific execution profiles and optimize based on those.

[0]: https://github.com/bytecodealliance/wizer

[1]: https://github.com/bytecodealliance/ComponentizeJS

That's not necessarily true. For example, Wasmtime uses Cranelift to compile WASM binary into native code. https://docs.wasmtime.dev/contributing-architecture.html

[1] https://pyscript.net

If you need/choose Pyodide load time will of course by much more but that's mostly because of the size of full core Python and the dependencies you might have.

> and NodeJS

Wait... what? Why?

For example, imagine you have a lib that converts markdown to html, but the lib happens to write the files directly to the disk, hence it can't be used in the browser. If you compile the nodejs runtime to wasm with a WASI that maps the file system to local storage, then you can just read the file from local storage after invoking the lib.

Just saying it is a technical possibility, this kind of approach is really only meant to be used if you _really_ just want to run some lib in the browser, no matter how slow it gets.

Also technically if you could compile JS to wasm (without nodejs, so more like how you run C through wasm right now) then you don't need to care about browser versions and JS api polyfills while still using JS.

Silliness aside, I actually had almost the same thought when I was reading the post lol. Great minds think for themselves so I guess maybe we're only really good minds then. Ahh well....

For those interested, Iron* is .NET.

IronPython

IronRuby

IronJS

https://github.com/FractalFir/rustc_codegen_clr

RustPython – A Python-3 (CPython >= 3.11.0) Interpreter written in Rust - https://news.ycombinator.com/item?id=35056586 - March 2023 (136 comments)

A full Python interpreter written in Rust - https://news.ycombinator.com/item?id=31086317 - April 2022 (7 comments)

Python interpreter written in rust reaches 10000 commits - https://news.ycombinator.com/item?id=29094323 - Nov 2021 (93 comments)

RustPython: A Python interpreter written in Rust - https://news.ycombinator.com/item?id=28280790 - Aug 2021 (49 comments)

RustPython – Python Written in Rust - https://news.ycombinator.com/item?id=27060802 - May 2021 (2 comments)

A Python interpreter rewritten in Rust, that can run pip - https://news.ycombinator.com/item?id=26030269 - Feb 2021 (48 comments)

A Python Interpreter Written in Rust - https://news.ycombinator.com/item?id=19064069 - Feb 2019 (194 comments)

Whenever I meet a C API issue on RustPython project (yes, I contribute to RustPython), I think about PyPy, and check what's going on C API, and realize 15 years was not enough to change things. Now the momentum of PyPy is not that strong as before. I believe Python community lost a huge chance.

Hope HPy or something can save PyPy. Maybe RustPython also will get a chance around there.

So much time lost by not embracing PyPy.

It is really underrated.

Maybe it's the extension issue, maybe it's something else or maybe it's the fact that it was born as an experimental platform more than anything

But it should have been more popular

No?

Python may dominate those fields vs other languages, but those fields don't dominate Python use, in my experience.

Jython stuck on Python 2 and doesn't have great interop with the rest of the world Python ecosystem, so I wouldn't recommend it for a new project, but it was a big force multiplier in that use case.

I have tested the latest version graalpy-community-23.1.2 as a regular Python interpreter from the command line (i.e., not through a Java program). It was able to run my standard library-only Python scripts, as well as a script that made async HTTP requests with the external library HTTPX. It couldn't run a TUI program that used the Python Prompt Toolkit (`AttributeError: module 'signal' has no attribute 'siginterrupt'`).

I am curious about your experience if you have used GraalPy more than I have.

I just now learned about this project, thanks for pointing it out. I'll be curious how they go about shared data, especially shared str/String.

also I personally happen to depend on Windows support, I'll go popcorn on the future of this project...

- https://www.pcgamer.com/inside-the-biggest-heist-in-eve-onli...

- https://www.mmorpg.com/news/an-eve-online-player-has-pulled-...

- https://www.escapistmagazine.com/the-infamous-eve-online-ban...

I haven’t seen much Jython since, but encountered a few enterprise tools doing this. It was an effective way of enabling advanced use cases and extensions of the existing Java stack without requiring the admin/developer to touch Java.

We switched to pypy several years back and that allowed us to push a great deal more batch jobs through per window with zero code changes.

Otherwise, for normal job running, cloud infrastructure makes this pretty tricky unless you're a big company. I work for a small company and we run python code on things like azure functions, until they support other python interpreters, we'd need a massive expected benefit to justify that sort of self-building of architecture.

I'm guessing almost nobody uses it for their SaaS, but outside of that, these runtimes do see some use.

[0] https://www.spotfire.com/

For anything that involves dependencies, I rely on venv or pyenv to create a clean environment. When on macOS, I tended to use docker/containers as well, but primarily because 99% of my Python work has been in a Linux environment and I wanted like-for-like as much as possible. But a good version manager directly on macOS should help quite a bit.

Same here. It's probably somewhat to do with my only partial MacOS/Homebrew knowledge, but every time I'd ressurect a Python project on MacOS, Homebrew would end up screwing up / confusing / munging the system Python and it's own ones.

Never ran into problems like that on Linux (except for that one time I tried Linux homebrew). Just something about the way homebrew does stuff seems incompatible with me understanding it. Seems to be a "just me" thing though.

I’m always having to dive in and figure out which packages are missing, wrong version, etc.

As I do this I find myself wondering if the repo maintainers make a habit of actually trying to set up their project from scratch just by following their readmes.

I’m assuming they just get out of date and then leave it to the community to troubleshoot their own installs.

For whatever reason I don’t seem to have nearly the same difficulties trying to clone and run rust or node projects for example.

More generally, the fact that there are about 20 different ways to manage your venvs, and that you seemingly tend to accrete every version of python released in the last 20 years with no clear way of managing all those installations, makes it quite confusing for newbies.

  git clone ...
  python -m venv venv
  source ./venv/bin/activate
  pip install -r requirements.txt

All the autotools/cmake/scons, library paths, header paths, PKG_CONFIG, etc. I've had so many issues building C projects that I can't even begin to remember all the issues.

> ...

> pip install -r requirements.txt

No. Usually that will pick up newer versions of the project's transitive dependencies, which will have breaking changes (because these days breaking changes in minor versions is what all the cool kids are doing). Since it's Python you won't find out until you hit the wrong codepath while using the program and get a runtime error.

To be fair, F# filled most of its niche after it "died" so it's not a need in any way, just a wish for nice things because there were a few years where IronPython was a very nice thing to have.

Note: I just realized that they mention wapm in their homepage. We phased out the WAPM CLI in favor of Wasmer (https://wasmer.io), so you can simply run in your shell:

  wasmer run rustpython

  import { Wasmer } from "@wasmer/sdk";

  let rustpython = Wasmer.fromRegistry("rustpython");
  let instance = await rustpython.entrypoint.run({
    args: "-c \"print(1)\""
  });
  let output = await instance.wait();

Perhaps this kind of thing needs to be baked into a language offering from the start?

Compare this to say, Rust, where the safety guarantees are useful in their own right.

There would be value in Rust even with zero packages, but I couldn't say the same for Python.

as for your 3rd point about packages, just try this: `grep -l '#!.*python' /usr/bin/*` and then run that same command piped into `xargs grep import`. i.e. measure how many applications on your system are importing anything besides the builtins/standard libraries shipped in the base interpreter.

for extra fun, try porting the matches above to Rust and see how fun that is without `use clap`/`use structopt` and `use envlogger`. i think you've got the last point completely backward, not that i think it's a particularly important measure of a language anyway.

The language is a joy and one of the main reasons it became popular. How do you think it got its ecosystem?

Personally, I find the lack of static types makes maintenance a nightmare, and think the build and deployment situation is miserable.

Python 2 is dead and that's why Jython and IronPython, who have failed to catch up with Python 3, are dead as well and are not worth mentioning, are they?

It's been working quite well, though lack of C extensions is a problem. We're hoping to move to CPython once the wasi and C extension support is there.

But the project works, compiles to wasm32-wasi, and can execute on the live ICP network: https://github.com/demergent-labs/kybra

    def fib(n):
        if n == 0 or n == 1:
            return 1
        return fib(n-1) + fib(n-2)

    print(fib(35))

    time python3 ~/code/fibs.py
    14930352

    ________________________________________________________
    Executed in    1.18 secs    fish           external
       usr time    1.14 secs  180.00 micros    1.14 secs
       sys time    0.01 secs  616.00 micros    0.01 secs

    time ./target/release/rustpython ~/code/fibs.py
    14930352

    ________________________________________________________
    Executed in   13.44 secs    fish           external
       usr time   13.32 secs  175.00 micros   13.32 secs
       sys time    0.02 secs  776.00 micros    0.02 secs

Most other Python implementations don't provide any way to use CPython C extensions.

There is also the HPy API and ABI, which some C extensions target and multiple Python implementations can compile or load.

https://github.com/hpyproject/hpy/pull/457

…and this one is no exception -> https://github.com/RustPython/RustPython/issues/1940

Packages that rely on c dependencies like numpy, etc. only work if you write a custom implementation by hand; the “normal” package flat out doesn’t (and cannot) work.

So basically you get no packages that use native extensions, unless the project explicitly implements support for them.

Pypy is the only implementation I’m aware of that has implemented a c api that is mostly compatible (see https://doc.pypy.org/en/latest/faq.html#do-c-extension-modul...)

For eg. Python running in a .net host or a Java host… I think this kind of compatibility would just be flat out impossible.

For rust… hm… probably possible. For wasm? Definitely not.

There are so many cases where you'd rather not give extension scripts unlimited access to the OS and file system, you'd think this option would be more common...

(except for some minor parts as well as the python in rust cross compilation case (last I tried ~1y ago), the rust in python cross compilation is fine tho)

Only issue I’ve had so far are some weird issues with conda versions and maturin. But those were basically my fault.

One of the things that turned me off of C#'s Blazor WASM was the payload size. I've found some of Rust's offerings like yew and leptos more interesting.

What would be a use case for this?

The code level integration (pybind11) was nice and easy, the issues came later. We found operational problems (multithreading), performance (in particular initialization) and worst: memory leaks and crashes on python VM teardown. Another problem was that python has it's own particular ideas about package management than generally don't rhyme with packages from the OS vendor.

I'm not sure what the angle is in this project, but I don't see a happy ending for projects that rely on both pip and cargo when it comes to maintenance and long-term support except for source-level integration. That is, unless there is some well designed and well thought out strategy to combine Rust and Python worlds that doesn't conflict with the RPM or apt view of doing things.

But I think, the point GP is making is more about the dependencies Python ads to your C++ (or Rust). Ie.: build RustPython project to produce artifact that works on my machine, but relies on my python interpreter, my (which may have C dependencies) as well as a rust edition and cargo. How can I bundle this into a package for a different Linux? Preferably do it following the existing packages and lore of that distro, rather some python and some rustup version?

Exactly! I want customers to be able to install my software on stock Ubuntu 24 / Rocky 9 / .. and seamlessly integrate with existing stock Apache, libcurl, libssl without any recompilation etc.

Having a bunch of pip dependencies tag along makes this quite complicated.

Lua being designed specifically for embedding makes it a good choice.

I wonder, though whether we might have ended up with similar fights maintaining LuaRocks dependencies in rpm/apt ecosystem on various platforms as soon as those customizations start pulling in dependencies like XML parsers, crypto, etc.

Do you have thoughts on that? (Note, we target linux, but sell proprietary closed source binaries only, except for the customizations)

I've spent tons of time tracking down python crashes from C extensions -- not how a company wants their devs spending their time -- and 99% of the time its just getting the reference counts right so C and python are on the same page. Dependency management of C pointers gets tricky sometimes...

My usual workflow when I'm wrapping a C/C++ library (which I do for 'fun' quite often) is to generate a skeleton of the python module with pybindgen and then hand-tune it until it works. I could write a bunch of custom wrapper code (which I do use when it makes things easier but it gets thrown away after the skeleton is generated so doesn't have to be very robust) and just use the output from pybindgen but that takes a lot more work unless the library falls into pybindgen's happy path. Plus, as I'm doing it for fun, I don't mind spending time to prettify the generated code and add some py-sugar. And the way pybindgen generates docstrings isn't the best so those would have to get handwritten either way.

Admittedly, I just do this kind of stuff as a hobby and industry has different goals so "pinch of salt" and all that.

A baking spatula makes a bad cooking spatula. It's pliable and can easily scrape the curved surface of a bowl, but for flipping a fried egg it is hopeless - in a pinch it will work - but it would be nice to have something rigid.

Further, describing it as making you "wince" is a recipe for a flame war bordering on deliberate bating. Just describe your issues with the language in more than a single charged word. You don't have to write an essay, just "I don't like the package management environment". That way people can have a constructive discussion.

Are you trying to say that all tools are the same? There are no bad tools?

Why is it naive for a painful tool to make you wince?

So, you're left with a choice of C++, Rust (what else, C#, possibly Zig and a few other).

Rust stops being a "puzzle" once you've written enough of it and you just know how to do things. It has one major disadvantage though - returning back to write in languages like C++/Python makes you cringe because they don't have basic language tools like proper sum types or traits that get things done.

The Rust memory management puzzles are not really complicated. They just get in the way.

> they don't have basic language tools like proper sum types or traits that get things done

Sum types and traits are not getting things done.

99% of whose time? Where does this estimate come from?

Definitely not my case, and definitely not the case for lots of programmers I know and/or work with. Python has its own place (for example, in DS/ML) and while I use it myself on daily basis for what's it's best at, statements like "you should be programming just about 99% of your time in Python" make no sense at best.

And, note, it's not only about speed/performance; after writing lots of Rust (or even F#), it's hard to get back to Python because of missing expressiveness and basic language tooling.

Finally, speaking of Python (CPython in particular), part of why it's so attractive is all the fast binary extensions libraries written for it... in C++/Rust, not Python.

> Sum types and traits are not getting things done.

Disagreed, sum types are exactly what gets things done, regardless of language performance and other features. This kind of logic exists everywhere. That's why languages with proper (or even semi-proper) sum types like Rust, F# are a lot easier to express 'business logic' in. And e.g. that's exactly why serialization/deserialization can get so weird in Python where values can be of multiple types and you have to invent crutches to work around that, to simulate sum types in this way or another.

No, not me, because I’m less productive in Python than in Rust, assuming the same level of final product quality, even excluding runtime performance.

> Turns out 99% of the time you don't need to parse terabytes or have μs-response times.

Performance is not the primary reason to use Rust. It has way more to offer than performance.

So much this. I've been doing mostly Scala (and some Haskell) for the last 10 years, and now I'm really enjoying Rust precisely because of its language features. Performance is just a (really) nice bonus.

Other bonuses include simple deployments (single binaries versus massive JVM and fat jars), low memory footprint, simple builds that just work (Cargo is an absolute joy), great concurrency, great ecosystem, etc. etc.

Re: your point, I'd still recommend learning Rust even if it's just solving last year's AoC, to see how various data structures, enums, traits, iterators etc work. Even if you don't end up using Rust on daily basis, it will probably affect the way you structure non-Rust code and provide you with a few neat design ideas.

Do I:

python? python3? pip install? pip3 install? python pip install? python pip3 install? python3 pip install? python3 pip3 install?

And then everyone mentions "oh just use venv" or "conda" or docker or... It just never ends or it never clearly explains what to do. And all I want to do is run a relatively simple script. Or course I would use a docker image to run a more complex app. But it's frustratingly hard to manage for people who don't use it often.

I'm from a java/Kotlin world where things aren't perfect either. But at least dependencies are clearly defined, and the only thing you need is Gradle that updates itself through a wrapper script if needed and pulls all dependencies. And a JVM/JDK which is also configured/installed quite clearly in a specific location.

The simplest way (in the sense of having the fewest components required) is this, using only built-in tools:

  $ python3 -m venv --upgrade-deps my-virtual-environment
  $ my-virtual-environment/bin/pip install whatever-third-party-package
  $ my-virtual-environment/bin/python3 my-script.py

Note that Debian's python3 package leaves out some parts of the standard Python distribution, but you can install the python3-full package for a complete installation.

  > py -3 -m venv --upgrade-deps my-virtual-environment
  > my-virtual-environment\Scripts\pip install whatever-third-party-package
  > my-virtual-environment\Scripts\python my-script.py

Its still new but works well. I'm transitioning to it from an unholy mess of pyenv, pip installs and other manual hacks.

If you're starting a new python project that is more than just a straightforward script I'd use Rye from the get go.

[1]https://rye-up.com/

but python? it between conda, pip, pip3, virtualenv and who knows what else. its hard to make something that I knwo will just work on everything.

https://peps.python.org/pep-0723/ might make life much simpler for single-script applications.

  python3 -m venv venv
  source venv/bin/activate
  pip install whatever
  echo "source path/to/venv/bin/activate && python3 path/to/app.py" > myscript
  mv myscript /usr/local/bin

printf '%s\n%s\n' "#!path/to/venv/bin/python3" "$(cat path/to/app.py)" > /usr/local/bin/myscript

Path/to/venv/bin/python path/to/app.py?

Neat trick though, gonna adopt it!

With Gradle, where do you define dependencies? All with version numbers in the `dependencies` block? In a separate file, declaring a hash map of versions and then a hash map of dependencies, and then putting the latter into `ext`, which magically propagates to just about everywhere in the build silently? In a separate file using some plugin that tries to replicate the lockfile functionality baked into most other dependency systems? In a separate TOML file, using the Catalog?

...and that's just about dependencies, I could go on for days about other Gradle features...

It's not any better than Python. You're just used to it. The same complexity is there in both Python and Kotlin, you just learned to cope with one better than the other.

(Gradle with Kotlin DSL and good IDE support is a little better, mostly because Kotlin devs are not as afraid of touching the build as when it's done in Groovy. It's still 2 orders of magnitude more complex than whatever you use for Python - unless you use zc.buildout or something like that.)

WASM would be an option, but overengineered for my case. Ruby (through Artichoke), or Python (through RustPython); but they come with the downside of introducing Ruby or Python. I haven't decided yet, but would prefer Lua for it's simplicity. Or JavaScript (a subset) for how easy it can be limited in scope.

Are there any (example) rust programs out there that have embedded scripting, or runtime plugins or addons in a scripting language?

https://crates.io/crates/rhai

https://crates.io/crates/dyon

https://crates.io/crates/duckscript

You can use the above links to browse crate "dependents", that is other crates using these dependencies if you are looking for example usage.

https://docs.rs/crate/starlark/latest

Though Lua probably has lots of good and feature rich examples as well.

Can't speak at all to Ruby or Python integrations.

In the end, I'd suggest it depends on your audience and what existing skills or biases they're likely to bring to the table.

My one warning is that I've used software that integrated JS and it hasn't aged well. From the Adobe products with ExtendScript to other, older and incompatible JS engines before ES5 even, don't get locked into something difficult to update and keep current.

How big are my binaries going to be if I embed a whole Python interpreter in it?

Hello world w/ rustpython interpreting Python[1]: 15459264 bytes

(Rust 1.75.0; rustpython 0.3.0; MacOS / Apple Silicon)

[1] https://github.com/RustPython/RustPython/blob/main/examples/...

Rust compiler can sometimes make better performance optimizations because there are more guarantees that code works as it is supposed to.

You can bypass these with C of course, but you need more manual (and less secure) operations.

Monomorphization is an example where that happens, in C if we're sorting and de-duplicating Geese, Customers and BugReports, we're either writing three separate functions dedup_geese, dedup_customers and dedup_bug_reports, or we're using function pointers and we incur the function call overhead when our functions get called. Ouch.

In Rust (or C++) the monomorphization step is going to turn sort & dedup for Geese, Customers and BugReports into separate functions†, and yet we only wrote the code once.

To some extent you can try to mimic this in C via the "X macro" strategy, but now you're not even just writing C any more, you're writing the macros and maybe running them through pre-processing and trying to understand if the result does what you meant, it's a pretty horrible way to work, so again you're discouraged from doing it.

† However the compiler may spot that actually the machine code implementation for say, BugReport and Goose is identical and so it only emits one in the eventual binary with the other just aliased - which may confuse a debugger and thus a human trying to debug it.

Actually, I would not mind a python interpreter in rv64 assembly (near 0 SDK) instead.