They always start off well, then inevitably there's a concept or key terminology that gets glossed over without sufficient explanation. .

"The random variable is described by the probability density function. The probability density function is characterized by moments.

The moments of the random value are expected values of powers of the random variable. We are interested in two types of moments"

This is where my journey ended this time. OK, so is that the expected value of the expontnent.of the random value orthe expected value of the random value to some power... and what makes the power of a randome value so special vice just operating on the randome value itself.

It's like these authors get tired of "dumbing" things down at random points in their chain of thought and just decide to skip over stuff. Or, perhaps they don't understand the underlying concepts themselves and simply can't explain them to others.

Who knows. Just frustrating. At least in Udemy you can write to instructors and ask questions. Unlike a book author who aren't paid to respond.

You make a good distinction between someone being paid to respond and not. A feedback loop helps out a lot here.

Failing that, you need to take this into your own hands. Honestly, you’re probably going to have to tell yourself a new story to get there. Maybe having empathy for the difficulty of teaching. Maybe it’s finding some inner drive. I don’t know. But you need to look at that paragraph, accept it’s insufficiently explained, and take responsibility for understanding it.

I’m not saying to read it over and over until you “get it”. (I don’t know why people try that, it’s kinda foolish). A simple strategy works most of the time. Read it until you find a word or phrase you don’t sufficiently understand. Maybe that’s “random variable”. Maybe it’s “probability density function”. Find an explanation for that (Wikipedia, ChatGPT, textbooks, videos). The fun thing is this algorithm is recursive. So you’ll likely run into something you don’t know again. That’s okay, just keep going. If it’s really tough, a lot of the value of a tutor is steering this depth first search.

Get each concept to the point you understand it very well for the problem at hand. You don’t have to know everything about PDFs, but don’t hand wave it either. After this process, you’ll be able understand this paragraph and continue.

This may take a while! If something is in a new domain, it’s usual for you to actually spend more time backfilling knowledge than on the main content itself. That may make it not worth it for you, but it’s not inherently bad. And the next time for something similar should be faster.

A random variable is a function X(w) taking (eg) real values. In your probability space you already have an ambient measure space and an ambient probability measure P which takes sets in the measure space to [0,1]. The pdf is then the function defined on sets P(invX(q)). invX is a set valued inverse.

Ok, consider coin flips. Then X takes each element of sample space either to 1 or -1. Set values inverse of 1 is the sets that map to 1. Then we get the ambient probability measure of them.

You don’t really have to cope with measure theory in full to take this tiny step.

https://gregorygundersen.com/blog/2020/04/11/moments/

I have no idea how I got through undergrad and graduate school without internalizing this concept that seems so foundational. Whacky

This is the problem I find with the 3blue1brown videos. They're pretty but I never get any understand from them. To people who already know they nod and see all the concepts they are already familiar with shown in a neat way but to some (like me) they don't generally get me to understand. Too many pre-reqs or someting

So relieved to learn that I am not the only one!

This is why I don't even bother to read through such tutorials. To understand Kalman filter one first need to understand basics of probability and then the importance of Gaussian distribution (Kalman filter mathematical derivation assumes that all the probab distributions involved are Gaussian). Then one notices that Gaussian distribution is uniquely defined when you know it first and seconds moments (yes, you cannot dance around introducing moments at some point). And then pretty nasty math follows :-) Kalman filter is not an easy thing. Rudolf Kalman claimed in one of the interviews that without his filter American landing on the Moon would not be possible.

Obligatory meme: how to draw an owl: https://www.memedroid.com/memes/detail/265779.

https://www.bzarg.com/p/how-a-kalman-filter-works-in-picture...

Literally four sentences later you would have found: * The first raw moment E(X) – the mean of the sequence of measurements. * The second central moment E((X−μX)2) – the variance of the sequence of measurements.

And if you had gotten as far as the part you quoted, you would have seen an extended example of why one is interested in means and variances.

That was left up to me to hunt down... which is fine I guess, but I certainly wouldn't say this is "from the ground up". At the very least, link to some external content that provides the necessary definitions.

Can anyone point to anything that explains it in a manner a programmer would explain (eg array iteration instead of sigma notation) ?

As far as i understand, Kalman Filters seem to be a moving average of position, velocity, and (perhaps) acceleration, and use those 3 values to estimate the 'real value' as opposed to what the sensor is telling you.

The Kalman filter is really just Bayes Theorem applied to a particular set of system dynamics. Really it’s two applications of Bayes rule: given my current state estimate, how do I update my estimate when I receive a measurement? Then, after receiving that measurement, how will the system dynamics impact my uncertainty before the next measurement? You iterate applying these two every time you receive a measurement.

Do you have a dynamics model? I.e., a set of equations that tell you how what ever you are trying to track will update in time? That’s the first thing you need and if you don’t have that, I could see you being quite confused about the implementation.

It might help you to either read about particle filters or just Bayes filters generally. The Kalman filter is a Bayes filter for a specific set of assumptions that leads to a lot of linear algebra. If you understand the Bayes filter, the Kalman filter will make more sense, and you can learn about the Bayes filter without all the matrices. The particle filter is another algorithm worth reading about. Like the Kalman filter, it is a Bayes filter but it doesn’t require the linear algebra.

Aside: I always get frustrated when I see people saying that you don’t need to know math to be a software developer. Maybe that’s true if you want to make web pages, but the more math you know, the better your ability to model problems and either solve them yourself or find solutions others have created. The scope of problems you can solve via programming becomes so much bigger the more math you know.

Sort of. The actual variables (v, r, a in your case) vary depending on your use case. What matters is you make an estimate based on that state, then you take a measurement and there’s an iterative way of adjusting your next state based on those two. It’s actually pretty easy to code once you understand formulas.

Maybe this could help http://bilgin.esme.org/BitsAndBytes/KalmanFilterforDummies

You need a linear discrete-time dynamical system model of your process. With the possible additions of external control action, and process noise. Your dynamical system has an internal state, described as a vector of values. Then you also need a linear model, of how the internal state translates to your observations, plus observation noise.

x – model state F – linear state transition model: x(t+1) = F x(t) Q – covariance matrix for process noise, so actually x(t+1) = F x(t) + N(0,Q) H – linear observation model R – covariance matrix for observation noise u, B – external control vector u, and B for how u acts on the model state. Optional.

N(0,Q) is a normal distribution with 0 mean and covariance Q.

For example, in the case of a moving physical object, the model state is

    [ x ]
    [ v ]

    [ 1 0 ]
    [ 0 0 ]

    [ 1 0 ] [ x ] = [ x ]
    [ 0 0 ] [ v ]   [ 0 ].

Then Kalman filter tells you, how to best estimate the real state vector at each time step, when both your state transition model, and your observation contains noise.

You are correct, many Kalman filter tutorials are a mess, where the explanation of the parts of the chosen process model, and parts of the Kalman filter, are blended together and the reader has a hard time telling them apart. And quite often they start with the simplest possible example, where the model state is 1-dimensional, i.e. just one number. And where the process model is a constant (no change): x(t+1) = x(t). So F = 1. This can lead even further confusion, what is the dynamical model and what are the parts of the Kalman filter, when the model is so small that it sort of disappears in the equations.

Better formatting for this part:

x – model state

F – linear state transition model: x(t+1) = F x(t)

Q – covariance matrix for process noise, so actually x(t+1) = F x(t) + N(0,Q)

H – linear observation model

R – covariance matrix for observation noise

u, B – external control vector u, and B for how u acts on the model state. Optional.

A moving average is a good place to start thinking about it. Think about a case where you would use a moving average, and why. You are probably using it because you have some measurement (sensor) that you know is noisy, and so by averaging out the noise (taking the mean of a number of samples) you try to get a better estimate of the true value. If you know how noisy the sensor is, you can get an idea of how many samples you should average over to get a good measurement. You can also take the standard deviation and report both the mean and variance of your measurement over multiple samples if you wanted.

For purposes of this example moving forward -- we are going to estimate all of our sensed or inferred values as a gaussian, parameterized by mean and variance. It's a simple way to give a measurement with some uncertainty around it.

This can be a good start if you know nothing little about the system you are measuring, other then the sensor / measurement is noisy. However for many systems you may have multiple quantities you are interested in estimating, and you have some idea of how they relate to each other.

Take your example of a physical system with acceleration, velocity and position. Basic physics will tell you that if at time t, you are at position p, moving at velocity v, then at time t+dt, your position should be roughly p+(v*dt). Similarly, you can update your velocity estimate using your estimate of acceleration. If this is a system under your control, then you can also take things like a force you commanded to update your acceleration model. This is great, by using physics, without any measuring after time 0, we can just figure everything out forward in time forever, by simply using our process model! However, because your initial estimates had some uncertainty, what you will find is that, if you just keep doing this, the uncertainty grows larger and larger with each time step, and eventually become so large as to be useless.

Enter tha kalman filter. What the kalman filter does is tries to combine the information given by your sensors and combine it with your process model to give you a better estimate of the quantities you are interested in than you could get from either technique alone.

Every time step the filter will make an state estimate using the process model based on your previous state estimate, and then use your current sensor measurements to update that state estimate, both in terms of the mean and uncertainty. In the basic kalman filter you assume your process model is linear, all of your estimates are simple gaussians, and then decide how much you want to weigh your model vs. your sensors via a simple multiplying factor, "the kalman gain"

Sorry again I couldn't write this out as a program, and the likely horrible run-on sentences that come from typing on a phone, but I hope that a quick overview of what the technique is trying to do will help make it a little easier to fill in the owl!

Once you get the intuition, Kalman filters are really interesting. As are particle filters -- those are fun to work with and visualize.

Care to elaborate?

Entering "never fired its guns in anger" in your favorite search engine should give several examples.

Here's one in an article from last week. https://owlcation.com/humanities/What-If-The-USS-Arizona-Nev... - "The USS Arizona never fired her guns in anger."

https://dictionary.cambridge.org/dictionary/english/in-anger

And yes. I'm an American.

I just searched Google for a comparison and found that EMA is as good as Kalman for "random walk plus noise" on a Stats Stack Exchange, and that a 2003 paper from Brown University (Joseph J. LaViola) showed that a Double Exponential Smoothing algorithm is of equal quality to Kalman (and extended Kalman) but 135 times faster, and a simpler approach.

I find Double Exponential Smoothing to be much easier to understand than Kalman, and assuming the LaViola paper is correct, I'm not going to put additional effort into understanding Kalman.

The advantage of particle filters is that they can handle complex scenarios with nonlinear physics and non-Gaussian distributions.

For example, a vehicle GPS unit could use street maps to eliminate impossible locations based on the recent history of turns. Gaussian filters can’t do that and just result in blurry blobs that cover several blocks.

Also l, what other types of newer models or more advanced models other than the particle filter would you recommend for offline processing?

Bayesian Filtering for Location Estimation (https://rse-lab.cs.washington.edu/postscripts/bayes-filter-p...)

It might help to first review importance sampling. https://en.wikipedia.org/wiki/Importance_sampling

A comprehensive survey (for 2003) is Zhe Chen's "Bayesian Filtering: From Kalman Filters to Particle Filters, and Beyond". A contemporary survey would surely incorporate machine learning.

https://people.bordeaux.inria.fr/pierre.delmoral/chen_bayesi...

For extensions see "An Overview of Existing Methods and Recent Advances in Sequential Monte Carlo"

https://people.eecs.berkeley.edu/~jordan/sail/readings/cappe...

"Let's begin with an intuitive example: think of a thermostat as it adjusts to the temperature. Got that example? Good! Next here is some advanced linear algebra to help make it more intuitive"

Has anyone comes across an explanation of Kalman filters that doesn't immediately dive into the math?

This looks to be the course playlist: https://youtube.com/playlist?list=PLAwxTw4SYaPkCSYXw6-a_aAoX...

The kalman filter stuff starts at video "Tracking Intro - Artificial Intelligence for Robotics"

Also the free course appears to be available here, although a login is required to access:

https://www.udacity.com/course/intro-to-artificial-intellige...

https://www.youtube.com/watch?v=QJSEQeH40hM

http://mlss.tuebingen.mpg.de/2013/2013/bishop_slides.pdf#pag...

This is half right: you don't learn math by reading proofs, you learn it by writing proofs. Developing strong intuition without proofs is possible (albeit difficult) for applied topics, but not viable at all for pure math.

It's only supposed to be somewhat intuitive for those who already possess some intermediate level college math background.

And no, rejecting the notion that not every advanced topic is supposed to be accurately ELI5-ed with a cringy Redditesque tone, means neither there's any gatekeeping taking place nor there's lack of understanding from the potential explainer's side.

But I've seen both it and the alternative. It is literally easier to learn the math, and then use that to facilitate communication, than to attempt to directly communicate key concepts while avoiding the math.

Is the Kalman filter a low-pass filter? Sometimes!

https://jbconsulting.substack.com/p/is-the-kalman-filter-jus...

PS: I've used it to remove jitter in virtual camera movement while cropping video around faces in real-time, streaming detected face locations to a Kalman filter worker and get back stream of stable camera locations.

[0]: https://www.mathworks.com/help/vision/ug/using-kalman-filter...

This is pretty complex but its been super helpful for me in terms of understanding PID controllers.

(A plant is a generic term used in controls for “the system you are controlling”)

[1] https://www.researchgate.net/profile/Mohamed-Mourad-Lafifi/p...

Beyond the scope of this comment and post, but mind two-sided filters (as the Kalman smoother/HP filter uses) as you could be incorporating future/unknown data into your model.

Briefly: you first run the filter equations "forwards", processing each datapoint sequentially from start to end. Then you run the smoother "backwards" in time on the same data going from end to start.

1. http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/...

The Kalman Smoother can be used to go back and update these past values with all the samples up to the current time. To update your previous measurements, you need to save the state of your filter at every timestep (x_k) and its associated covariance matrix (P_k). You can then apply Kalman Smoothing to reprocess previous data and update it with all current information. This will often remove the phase delay that you would otherwise observe in your estimate.

Tl;dr: It's mathmatically equivalent to the Bayes filter equations (no big surprise, KF is a BF), and those equations are easier to understand intuitively, but less efficient to compute.

In general, using Bayes theorem for filtering is often called the Bayes filter. Outside of a limited set of circumstances, it is not possible to implement. The Kalman filter is one set of circumstances because linear transforms of Gaussian distributions remain Gaussian, and we can specify a Gaussian process just by its mean and covariance. Another set of circumstances is when you have a discrete system with a small enough number of states.

In general however, the Bayes filter results in an infinite dimensional dynamical system since it yields a typically continuous function representing the distribution of your state given your measurements. The Kalman filter, particle filter, and other methods approximate this problem via a more tractable finite dimensional representation.

https://www.youtube.com/watch?v=FkCT_LV9Syk&list=PLzCl0zqNaI...

I think that this website needs to improve the experience of mobile users. First, the book index on the landing page didn't contain hyperlinks to the free book chapters and these chapters don't appear on the preview, so users may assume it is a shady marketing tactic (trial, requires retweet, subscription to book platform, ...). Second, towards the end of the page, there could be a one-sentence-description of the next section with an hyperlink. Right now there is only a "next" button that is easy to miss, and it can be confused with generic "next post" buttons used by many blog platforms that users learn to ignore because they tend to be useless.

How is this different than any of the ML models?

Where a KF is really going to kick the pants of a multi-layer perception/neural network is how computationally efficient it is. A KF only takes a couple of matrices of size N^2, where N is the number of variables you’re trying to predict. Compare this to a NN with hundreds/thousands of nodes. Also, the KF is “online learning” in that it “is trained as you go” rather than some other ML models that require upfront training, a KF is very useful for live update-and-predict use cases. And, again, it’s extremely computationally efficient, and can run easily on embedded systems. The book ad, here, suggests tracking: so tracking an airplane with an air traffic control radar would be an effective use for a KF. (Where NNets have found any other uses I’m sure you’re aware of.)

Another huge benefit of a KF is that, unlike NNets, KFs are “explainable”, and in fact extremely well understood by many professionals. This means that a KF can be better tuned to suit a purpose with less fear of unexpected results that may be more common in other ML models. Like, KF(S, x) will always return an explainable new state, where NN(x) may result in a surprise state and no amount of analysis can reveal why (and require training a new model, the “retrain and pray” solution).

There’s a couple of differences for you.

A lot of technical decisions aren't based on "what's the quickest, cheapest and easiest solution to the problem?" but "what solution is most likely to get me hired at a pay bump when I jump ship?"

Kalman Filters are used in the context of a very specific model-type for dynamic systems (a state-space model, see below) to update states (xₖ) using feedback data from sensors (yₖ). These state-space models can either be derived by fitting data, or they can be derived from first principles through physics equations.

  xₖ₊₁ = f(xₖ) + g(uₖ)

  yₖ = h(xₖ)

For instance, when driving a car, examples of states (x) are position/velocity/acceleration (which might not be directly measured with a sensor! But can be backed out from a mathematical model from quantities that are measured), sensor measurements (y) might be speedometer, accelerometer readings, and control actions (u) might be throttle position, brake pressure, steering angle. The Kalman filter has a model relating all this in time, and based on that model and sensor readings, it reconstructs/infers the likeliest states in the presence of even noisy measurements. This is why Kalman filters are known as "state estimation" algorithms.

ML models typically do not do this -- they only predict. Kalman filters predict and update.

You basically need to know some kind of a model for the system to run KF. Whereas ML is all about working out the model automatically.

As for similarities, KF is a really efficient implementation of Bayesian inference. I think that any ML model that isn't fundamentally using Bayesian inference, is fundamentally flawed.

ML requires training, significant amounts of compute power, and large datasets.

The Apollo program used Kalman filters with limited compute resources.

Kalman filters are for predicting system states in the presence of uncertainty; ML is really searching for and matching patterns, under uncertainty not in its training set, it tends to to find the glitch in the matrix.

In other words it performs well for certain applications specifically because it allows you to bring in domain knowledge in the form of the process model and known uncertainties. Whereas deep learning models try to generalize the model and learn implicit structure from data.

The model is updated sequentially (online learning).

https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80...

Actually in theory you can replace everything with it. So what's the point of asking this question here? Ask it for everything.

Kalman filters, and other similar digital filtering and prediction algorithms, are like scalpels compared to the broadsword of NNs and such. There are plenty of things that you can't or shouldn't use a kalman filter for, but for the tasks that it is suited for, you cannot do better with another solution. ML is mostly hand wavy bullshit, and DSP algorithms are like... doing real math, real engineering.

Two of (arguably the best) open source RC aircraft flight controllers (ArduPilot and PX4) are using extended Kalman filters in their state estimators (essentially sensor fusion that provides attitude/position estimate):

https://github.com/ArduPilot/ardupilot/tree/master/libraries...

https://github.com/PX4/PX4-Autopilot/blob/main/src/modules/e...

I'm not that familiar with cleanflight/betaflight/inav scene to know what the FPV racer flight controllers use.

They are a simple tool, and simple often works. You just need to know the limitions and know which variant (if any) is best applicable to the problem at hand.

It is just that I keep seeing KF be mentioned on Twitter and blogs etc. for using it in finance.

I know someone with a PhD in control systems, works for an eVTOL startup, and their title is something like "estimation expert" or "estimation specialist". That is, 100% of their job is designing relatively complex KF-like algorithms that are used to estimate the aircraft state (position, pose, velocity, wind conditions, etc) in real time based on a bunch of sensors.

Often some tweaks from the standard formula are necessary to account for real-world non-linearities, and some creative design work is required to define states in such a way that the Gaussian noise assumption can hold well enough.

Yes, ship/vessel navigation software heavily use Kalman Filters. Especially on the inputs received from the various position reference sensors.

We use a type of Kalman Filter to estimate direction and instantaneous dynamics of the drill string, this way the drilling operation no longer needs to stop to get a directional measurement.

We track targets (mostly ships, mostly maritime, but not always) and we use Kalman Filters (and variants) extensively.

https://en.wikipedia.org/wiki/Kalman_filter#Nonlinear_filter...

Contrary to your experience, there was a time when we were ridiculed for not using Kalman Filters, but in the limited niche we inhabited then, our internally developed algorithms out-performed Kalman.

But mostly, these days, yes, we use Kalman Filters of various types.

Could you tell me more about this? What other algorithms are used for position tracking and motion estimation. I have seen various ML models... RNN/DNN used. I'm guessing with VTMIS you are doing time-series predictions?

8-(

[citation needed]

Siemens (SIMATIC PCS 7)

https://support.industry.siemens.com/cs/document/109748837/s...

Bosch Sensor Fusion

https://www.bosch-sensortec.com/products/smart-sensors/bha26...

Kalman filters are not state-of-the-art, but they are fairly simple to implement and are still used.

https://www.patentlyapple.com/2020/11/apple-reveals-adding-t...

I'm 100% sure there's some form of EKF on some loop on AirTags too.

The naive Kalman filter is only suited for linear problems; extended and unscented Kalman filters (EKF/UKF) are necessary for anything non-linear (including rotation). In any case, they build on the basic KF, so you have to understand that first.

If this is the current state of the art, are there generally-available/open-source libraries existing that implement this and practitioners use for this?

The only one I could find is https://github.com/kartikmohta/manifold_cdkf, which currently has 8 Github stars.

I also found an approach mentioned in [2] that is to just treat a single rotation angle as linear, and then wrap it around at 180 degrees in between state updates with additional conditional logic. Is this what people did in practice before? I cannot find substantial info on this.

How did people use KF on physical objects before 2010?

[2]: https://old.reddit.com/r/ControlTheory/comments/d2yrjq/kalma...

The authors of [1] do discuss and link to their own library in section 5.

However, in my experience, most people implement the math themselves rather than use any libraries (beyond e.g. Eigen).

> How did people use KF on physical objects before 2010?

The Multiplicative EKF (MEKF) was used since 1969 according to [3]. It's a hacky approximation of [1]. [1] is really just a generalization/unification of lots of application-specific hacks that were used before, including the MEKF.

[3]: https://ntrs.nasa.gov/api/citations/20040037784/downloads/20...

> All People I have interacted with have never seen Kalman Filters outside Academica. Do anyone here have industry experience where Kalman Filters is actually used in prod?

Looking at prior academic work in an academic paper won't help answer this question.

If you model your rotation as a quaternion, there is a way to linearize the update process of the quaternion and use a KF. This can work very well and is what most quadrotors I’ve worked with do. However, care is needed to ensure the result gives a valid updated rotation and that you implement the disgustingly messy equations correctly.

Side story - a friend of mine tried to recruit me to a well known robotics startup in the commercial autonomous air vehicle space (DM me if you’re curious who). The technical portion was basically, “implement a state estimator for radar target tracking, In one day, here’s some data”. Meanwhile, I have a day job. Lol, I told ‘em, “I’m not a good fit if this is what you’re looking for”. Oh yeah, my recruiter/friend also said they were looking for rock stars. I feel like I really dodged a bullet there.