Nice writeup. F1, balanced accuracy, etc. In truth it depends on your problem and what a practical "best" solution is, especially in imbalanced scenarios, but Matthews Correlation Coefficient (MCC) is probably the best comprehensive and balanced blind go-to metric, because it guarantees that more portions of the confusion matrix are good [0,1].

I made a quick interactive, graphical exploration to demonstrate this in python [2].

[0]: https://biodatamining.biomedcentral.com/articles/10.1186/s13...

[1]: https://biodatamining.biomedcentral.com/articles/10.1186/s13...

[2]: https://www.glidergrid.xyz/post-archive/understanding-the-ro...

MCC also generalizes to multi-class well. I wish it had a better name though. It seems like F1 score has better marketing

Really neat visualization! And thanks for the tip on MCC.

Out of curiosity I plugged it to the same visualization (performance vs. class weight when optimized with BCE) and it behaves similar to F1, i.e. best without weighting.

The only thing that matters is your estimation of how the balance will change out of distribution or with future data etc

Insane how tricky imbalanced stuff gets- I always end up second guessing my metrics tbh. You think there's ever such thing as a "right" number for real world junk or is it just endless tradeoffs?

Yeah it gets tricky. I think eventually it has to be about tradeoffs - no ML system can be 100% correct. I do think there's a "right" decision (up to a point) in the context of the product or business.

I read the article and the take away is that class weights and stratified sampling did not help for the OPs problem.

[5/11, 1:41 PM] Meta AI: Imbalanced learning refers to the challenge of training machine learning models on datasets where the classes are not represented equally. This can lead to biased models that perform well on the majority class but poorly on the minority class.

Class Weight Class weight is a technique used to address class imbalance by assigning different weights to each class during training. The idea is to give more weight to the minority class and less weight to the majority class.

Key Considerations 1. Class weight calculation: Weights can be calculated based on class frequency or other heuristics. 2. Hyperparameter tuning: Class weights can be tuned as hyperparameters during model training. 3. Evaluation metrics: Metrics like F1-score, precision, and recall are often used to evaluate model performance on imbalanced datasets.

Techniques 1. Oversampling: Oversampling the minority class to balance the dataset. 2. Undersampling: Undersampling the majority class to balance the dataset. 3. SMOTE: Synthetic Minority Over-sampling Technique (SMOTE) generates synthetic samples of the minority class.

Applications 1. Fraud detection: Imbalanced learning is crucial in fraud detection, where the minority class (fraudulent transactions) is often much smaller than the majority class (legitimate transactions). 2. [5/11, 1:41 PM] Meta AI: Imbalanced learning refers to the challenge of training machine learning models on datasets where the classes are not represented equally. This can lead to biased models that perform well on the majority class but poorly on the minority class.

Class Weight Class weight is a technique used to address class imbalance by assigning different weights to each class during training. The idea is to give more weight to the minority class and less weight to the majority class.

Key Considerations 1. Class weight calculation: Weights can be calculated based on class frequency or other heuristics. 2. Hyperparameter tuning: Class weights can be tuned as hyperparameters during model training. 3. Evaluation metrics: Metrics like F1-score, precision, and recall are often used to evaluate model performance on imbalanced datasets.

Techniques 1. Oversampling: Oversampling the minority class to balance the dataset. 2. Undersampling: Undersampling the majority class to balance the dataset. 3. SMOTE: Synthetic Minority Over-sampling Technique (SMOTE) generates synthetic samples of the minority class.

Applications 1. Fraud detection: Imbalanced learning is crucial in fraud detection, where the minority class (fraudulent transactions) is often much smaller than the majority class (legitimate transactions). 2. Medical diagnosis: Imbalanced learning can be applied to medical diagnosis, where the minority class (diseased patients) may be much smaller than the majority class (healthy patients).

Would you like to know more about imbalanced learning or class weight?