展示HN:使用朴素贝叶斯算法的Go语言垃圾邮件分类器
Show HN: Spam classifier in Go using Naive Bayes

原始链接: https://github.com/igomez10/nspammer

## Go语言实现的朴素贝叶斯垃圾邮件分类器 这个Go包,`nspammer`,实现了一个基于文本的垃圾邮件检测的朴素贝叶斯分类器。它利用贝叶斯定理,并带有朴素独立性假设,并结合拉普拉斯平滑来防止遇到未见词时出现零概率问题。 该分类器在带标签的消息数据集(垃圾邮件/非垃圾邮件)上进行训练,并提供了一个简单的API来对新的文本输入进行分类。在训练期间,它计算垃圾邮件和非垃圾邮件类别的先验概率和词频。 分类涉及计算对数概率,以确定消息更有可能被识别为垃圾邮件还是非垃圾邮件。该包包含使用简单示例和Kaggle垃圾邮件数据集的测试,以便在真实数据上评估准确性。它可以通过`go get github.com/igomez10/nspammer`获得。

一位Hacker News用户分享了一个基于Go语言的垃圾邮件分类器链接,该分类器利用了朴素贝叶斯算法,发布在GitHub上。 这篇文章引发了讨论,一位评论者分享了十多年前的一个类似Perl实现的版本,并提到计划加入向量化功能。 对话涉及开源项目许可的重要性,以及一篇Paul Graham关于构建更好软件的文章链接。 有人担心LLM生成垃圾邮件的兴起可能会降低贝叶斯垃圾邮件过滤器的未来有效性,但其他人认为LLM独特的模式应该可以检测到。 最后,用户回忆了2010年代贝叶斯统计在工程师中流行的一个时期,尤其是在A/B测试中,一些人认为这受到社会学和像Eliezer Yudkowsky这样的人物的影响。 讨论还强调了贝叶斯统计在特定A/B测试场景中优于频率学方法的优势。
相关文章

原文

A Naive Bayes spam classifier implementation in Go, enabling text classification system using the Naive Bayes algorithm with Laplace smoothing to classify messages as spam or not spam.

  • Naive Bayes Classification: Uses probabilistic classification based on Bayes' theorem with naive independence assumptions
  • Laplace Smoothing: Implements additive smoothing to handle zero probabilities for unseen words
  • Training & Classification: Simple API for training on labeled datasets and classifying new messages
  • Real Dataset Testing: Includes tests with actual spam/ham email datasets
go get github.com/igomez10/nspammer
package main

import (
    "fmt"
    "github.com/igomez10/nspammer"
)

func main() {
    // Create training dataset (map[string]bool where true = spam, false = not spam)
    trainingData := map[string]bool{
        "buy viagra now":           true,
        "get rich quick":           true,
        "meeting at 3pm":           false,
        "project update report":    false,
    }

    // Create and train classifier
    classifier := nspammer.NewSpamClassifier(trainingData)

    // Classify new messages
    isSpam := classifier.Classify("buy now")
    fmt.Printf("Is spam: %v\n", isSpam)
}

NewSpamClassifier(dataset map[string]bool) *SpamClassifier

Creates a new spam classifier and trains it on the provided dataset. The dataset is a map where keys are text messages and values indicate whether the message is spam (true) or not spam (false).

(*SpamClassifier).Classify(input string) bool

Classifies the input text as spam (true) or not spam (false) based on the trained model.

The classifier uses the Naive Bayes algorithm:

  1. Training Phase:

    • Calculates prior probabilities: P(spam) and P(not spam)
    • Builds a vocabulary from all training messages
    • Counts word occurrences in spam and non-spam messages
    • Stores word frequencies for likelihood calculations
  2. Classification Phase:

    • Calculates log probabilities to avoid numerical underflow
    • Computes: log(P(spam)) + Σ log(P(word|spam))
    • Computes: log(P(not spam)) + Σ log(P(word|not spam))
    • Returns true (spam) if the spam score is higher
  3. Laplace Smoothing:

    • Adds a smoothing constant to avoid zero probabilities for unseen words
    • Formula: P(word|class) = (count + α) / (total + α × vocabulary_size)
    • Default α = 1.0

The project includes support for the Kaggle Spam Mails Dataset. To download it:

This script requires the Kaggle CLI to be installed and configured.

Run the test suite:

The tests include:

  • Simple classification examples
  • Real-world email dataset evaluation
  • Accuracy measurements on train/test splits
联系我们 contact @ memedata.com