Meta 的 Muse 非常适合网页抓取。
Meta's Muse is fantastic for web scraping

原始链接: https://sigh.dev/posts/metas-muse-is-fantastic-for-web-scraping/

作者并未将 Muse 用于点餐、安排预约等日常任务,而是把它作为一个低成本、高吞吐量的 AI 智能体,用于名为 fullsets.fm 的业余项目。该网站汇总专业录制的音乐会,按艺人分类,并提取演出曲目单。Muse 会扫描 Reddit 和 YouTube,寻找冷门频道以及包含完整、高质量演出的视频;随后,它会帮助匹配艺人,并在视频获批前检查 Luna 和 Gemini 流程的处理结果。 作者每月花费 80 美元,让 Muse 执行大规模搜索任务,每周消耗数十亿个 token,而且用量似乎没有上限。Muse 的写作和编程能力一般,界面及任务执行也偶有不稳定,但它很适合重复性的 Gemini Flash 类自动化工作。本文还提出了一个问题:持续不断地大规模抓取网站,可能会促使更多平台屏蔽 Muse。文章中包含一个推荐码,以及一张受《人生切割术》启发的 AI 生成图片。

一则 Hacker News 讨论聚焦于 Meta 的 Muse,以及一套定制的“智能体浏览”框架。它通过旁加载的扩展和 WebSocket 控制一个独立的 Chromium 实例,执行滚动、点击、输入等常规操作,并通过高度模拟真人浏览器行为来通过 CAPTCHA 和反机器人检查。 技术评论者讨论了如何避开 Chromium 暴露的 `navigator.webdriver` 标志、清除自动化痕迹、随机化浏览器指纹、使用住宅代理,以及远程调试在连接建立前或建立后是否能够被检测。Muse 的固定 IP 地址段可能已经被封禁,从而降低其实际用途。 更广泛的争论涉及伦理和规模问题。一些人认为,抓取网页或解决 CAPTCHA 本身并非错误;另一些人则强调,这可能给小众社区带来意外伤害,因为大量未经披露的智能体流量可能压垮这些社区。评论者预测,网站会采取更严格的限流措施、会话数量限制、登录墙、IP 信誉系统、指纹识别和设备证明等防御手段。批评者警告,这些措施也可能给正常用户带来负担、限制开放访问,并加速形成一个更具敌意、对用户不友好的网络环境。
相关文章

原文

I haven’t found much use for the personal assistant part of Muse. People on X (Twitter) keep using these assistants to order DoorDash or something. I order DoorDash like five times a year, so that’s not going to speed anything up for me. I also don’t get many emails, and managing my appointments is easy enough already.

What I do need is a bunch of data from Reddit and YouTube and a boatload of cheap AI tokens. That’s where Muse comes in handy. I’m building fullsets.fm, a side project that aggregates professionally recorded concerts (festival livestreams, Tiny Desk, etc.), organizes them by artist, and extracts tracklists. I’m looking for complete performances with good audio and video.

I’ve got a pipeline that uses Luna and Gemini 3.8 via Gemini Studio to cheaply determine whether a YouTube video has the quality and content I’m looking for. What I don’t have is a good way to find the videos in the first place. I can scan a single YouTube channel for qualifying videos, but I need a way to find all the weird little channels, like T-Mobile Europe, that have for some reason uploaded a few random concerts. There are high-quality concerts scattered across all these little channels, and that’s part of why I built fullsets.fm (a work in progress).

Muse helps me find those channels and videos. I’ve also got it at the other end of the pipeline, helping with artist matching, reviewing what Luna and Gemini output, and making a final decision to approve or reject each video.

I can tell Muse to scan Reddit and YouTube for 1,000 videos matching my criteria, or build a list of artists who have ever played at Coachella and then go through them one artist at a time, searching YouTube for matching concerts. It sits there and spins until it’s done, punishing those websites with a machine in the cloud that has seemingly endless bandwidth and a decent CPU.

I paid $80 for a month of Muse, and I’ve had it use a few billion tokens a week on this work. A lot of the time, it also seems not to count token usage at all. Their model (Muse Spark 1.3) is pretty mediocre at writing, and I would never use it to code, but I’m finding it really useful for this Gemini Flash-style workload. They seem happy to give me effectively unlimited tokens and a managed VM with Chrome.

I worry about what will happen when someone points Muse at my own websites. These agents don’t sleep, and they can pretend to be real users extremely well. If it’s not fast enough, you can spin up more processes.

Anyway, I see why Amazon blocked it already. Let’s say my goal was dropshipping or something. Could I scrape thousands of Amazon pages all day long? I don’t know where this leads, but we’ll likely see more websites block Muse unless Meta prevents abuse. There are still some early rough areas of Muse, jobs will freeze in the middle, communication with this somewhat shitty model can sometimes be frustrating and I’m not sure the UI makes much sense when running multiple jobs that are using the browser.

If this post makes you want to try it out, my Muse referral code is 69WSDQ. Enter it in Settings within 48 hours of signing up and we both get a billion tokens.

A retro computer classifying concert recordings in an empty office
AI-generated image inspired by Severance
联系我们 contact @ memedata.com