https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid...

https://www.lesswrong.com/posts/LAxAmooK4uDfWmbep/anomalous-...

Vocabulary isn't infinite, and GPT-3 reportedly had only 50,257 distinct tokens in its vocabulary. It does make me wonder - it's certainly not a linear relationship, but given the number of inferences run every day on GPT-3 while it was the flagship model, the incremental electricity cost of these Redditors' niche hobby, vs. having allocated those slots in the vocabulary to actually common substrings in real-world text and thus reducing average input token count, might have been measurable.

It would be hilarious if the subtitle on OP's site, "IECC ChurnWare 0.3," became a token in GPT-5 :)

In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence indicates that.

Three logicians walk into a bar. The bartender says "what'll it be, three beers?" The first logician says "I don't know". The second logician says "I don't know". The third logician says "Yes".

Both of the first two logicians wanted a beer; otherwise they would know the answer was "no". The third logician recognizes this, and therefore knows the answer.

This way is logically most efficient to work and involve the least communication.

() In a lecture by the mathematician & author Sarah Hart.

Fully agreed that "hallucination" is a bonkers word for it — sensational and melodramatic. But few people know what a confabulation is, and moreover it's an overly complex way to describe the phenomenon.

The LLM is making something up. It's a fabrication.

It's not fanciful; it's not spooky; it's mundane, as it should be.

I believe that you can also cause something a bit like a transient dysphasia by giving them bad inputs as well, so there is that on the language production side. However there's still nothing that pertains to the experience aspects central to what hallucinations actually are.

Even if you don't show children videos, but want to play some music, YouTube is still the least-hassle, least-bullshit music stream player (arguably still it's main use for adults, too). Ain't anyone got time to deal with Spotify's ever more broken app. And this is the limit of technical skill of almost all parents. They can't exactly run SponsorBlock in YouTube's mobile app (and paid YouTube doesn't help here either, surprise surprise).

Not making excuses (though I'm not really blaming parents for this) - just saying how things actually are.

I used to cut all those things to shape with youtube-dl and Audacity; we have a library of a good hundred+ of sanitized songs to play, but with modern world hating files and anything offline, it turned out to be quite a hassle to keep the practice up.

If you have to put a show on TV to give some songs to sing along to or to distract them while you're making lunch, I'm not judging you, and I think it's best to put this content on a gradient rather than black and white.

Parenting is easier if you have 5 family members in walking distance and they also have similarly ages kids who can all play together.

At some point fairly recently they added a "I don't know the answer" button to the email, but it's much less prominent than the main call-to-action.

A lot more of LLM hallucination is it getting the context confused. I was able to get GPT4 to hallucinate easily with questions related to the distance from one planet to another, since most distances on the internet are from the sun to individual planets, and the distances between planets varies significantly based on their locations in cycle. These are probably slightly harder to fix.

I've noticed that while this can help to prevent hallucinations, it can also cause it to go way too far in the other direction and start telling you it doesn't know for all kinds of questions it really can answer.

What I'm pushing at is not that this linguistic ability naturally leads to the LLM behavior we're seeing and calling "hallucinating", just that LLMs may capture some of how humans process language, differentiate semantics, recall terms, etc, but without the mechanisms that enable rationally grappling with the resulting semantics and propositional (in)coherency that are fetched or generated.

I can't say this is very surprising—most of us seem to have thought processes that involve generating and rejecting thoughts when we e.g. "brainstorm" or engage in careful articulation that we haven't even figured out how to formally model with a chatbot capable of generating a single "thought", but I'm guessing if we want chatbots to keep their ability to generate things creatively there will always be tension with potentially generating factual claims, erm, creatively. Further evidence is anecdotal observations that some people seem to have wildly different thresholds for the propositional coherence they can spot—perhaps one might be inclined to correlate the complexity with which one can engage in spotting (in)coherence with "intelligence", if one considers that a meaningful term.

I don't think Wittgenstein would agree, first of all, that there is a "natural logic" to language. At least in the PI, that kind of entity--"the natural logic of language"--is precisely the kind of weird and imprecise use of language he is trying to expose. Even more, to say that such a logic "allows" for anything (like metaphors) feels like a very very strange thing for Wittgenstein to assert. He would ask "what do you mean by 'allows'"?

All we know, according to him (in the PI), is that we find ourselves speaking in situations. Sometimes I say something, and my partner picks up the right brick, other times they do nothing, or hit me. In the PI, all the rest is doing away with things, like our idea of private language, the irreality of things like pain, etc. To conclude that he would make such assertions about the "nature" of language, of poetry, whatever, seems like maybe too quick a reading of the text. It is at best, a weirdly mystical reading of him, that he probably would not be too happy about (but don't worry about that, he was an asshole).

The argument you are making sounds much more French. Derrida or Lyotard have said similar things (in their earlier, more linguistic years). They might be better friend to you here.

Dunno. GP?

Because expecting a behaviour, like knowing you don't know, that isn't represented in the training set is silly.

Kids make stuff up at first, then we correct them - so they have a way to learn not to.

The problem is that curating data is slow and expensive and downloading the entire web is fast and cheap.

See also https://en.wikipedia.org/wiki/Cyc

Maybe if you trained a small base model to know it doesn't know in general and THEN trained it on the entire web with embedded not-knowing preserving training examples, it would work?

This is not a brain. The best analogy is an English major.

They are good at language, not reasoning.

Humans see language and think reason. It seems we can’t separate the two.

Google has one of these already, with an LLM that was trained on nothing but weather data and so can only give weather-data-prediction responses.

The 'knowing it doesn't know things' part is much harder to get reliable, though.

Probably true, but if you have quality, organized data, you will just want to search the data itself.

I mean, it's inherent to LLMs to be unable to answer "I don't know" as a result of not knowing the answer. An LLM never "doesn't know" the answer. But they'll gladly answer "I don't know" if that's statistically the most likely response, right? (Although current public offerings are probably trained against ever saying that.)

On the same lines as why people argue if a tree falling in a wood where nobody can hear it makes sound because some people implicitly regard sound is the qualia while others regard it as the vibrations in the air.

How often do the words "I don't know" get uttered in books, papers, articles, stack overflow, or any other resource of knowledge?

An LLM should have no problem replying "I don't know" if that's the most statistically likely answer to a given question, and if it's not trained against such a response.

What it fundamentally can't do is introspect and determine it doesn't have enough information to answer the question. It always has an answer. (disclaimer: I don't know jack about the actual mechanics. It's possible something could be constructed which does have that ability and still be considered an "LLM". But the ones we have now can't do that.)

Answering "I don't know" because it a likely response to a particular string is completely different from being aware that one does not know the answer and saying so.

Both motivations lead to the same outcome, but they're unrelated processes. The response "I don't know" can represent either:

1. The most likely answer to a particular question, based on statistical data; or

2. An expression of an agent's internal state.

Figuring out that distinction is perhaps one of the most important questions ever raised.

This isn't true. There are many contexts where it is true but it doesn't actually generalize they way you say it does.

There are plenty of cases where experts in a non-one-on-one context will express a lack of knowledge. Sometimes this will be as part of making point about the broader epistemic state of the group, sometimes it will be simply to clarify the epistemic state of the speaker.

However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.

Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (because even the homepage is disallowed). Which means they wouldn't follow any of the links to other subdomains.

And then all the links are to external domains, which aren't subject to the first site's robots.txt

Although the crawler should probably ignore all the html body. But it does feel like a grey area if I accept your first pint.

I'm sure the big players like Google would deal with it gracefully.

  http {
    limit_req_zone $binary_remote_addr zone=ten_bytes_per_second:10m rate=1r/m;
    server {
      location / {
        if ($http_user_agent = "mimo") {
          limit_req zone=ten_bytes_per_second burst=5;
          limit_rate 10;
        }
      }
    }
  }

So 1.2 million non robots.txt requests, when his robots.txt file is configured as follows

    # buzz off
    User-agent: GPTBot
    Disallow: /

For those bad crawlers, which I very much disapprove, "not respecting robots.txt" equals "don't even read robots.txt, or if I read it ignore it completely". For them, "respecting robots.txt" means "scan the page for potential links, and after that parse and respect robots.txt". Which I disapprove and don't condone.

"You're disallowed, but go head and slurp the content anyway so you can look for external links or any indication that maybe you are allowed to digest this material anyway, and then interpret that how you'd like. I trust you to know what's best and I'm sure you kind of get the gist of what I mean here."

In this case, as in many, the disallow rules are intentionally meant to protect the signal quality and efficiency of the crawler.

Notice how he casually drops a link to the landing page in the NANOG message. That's how the bots will get a bait.

  # silly bing
  #User-agent: Amazonbot          
  #Disallow: /

  # buzz off
  #User-agent: GPTBot
  #Disallow: /

  # Don't Allow everyone
  User-agent: *
  Disallow: /archive

  # slow down, dudes
  #Crawl-delay: 60

It’s functioning exactly as designed.

Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..

If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.

This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view).

I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.

A tarpit is different because it's designed to slow down scanning/scraping and deliberately waste an adversary's resources. There are several techniques but most involve throttling the response (or rate of responses) exponentially.

It impacts the performance for the other legitimate users of that web farm ;)

The irony of the whole thing is brutal.

Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.

> The fears that we have “used all the data” are incredibly unfounded.

The problem isn't whether we used all the real data or not, the problem is that it becomes increasingly difficult to distinguish real data from previous LLM outputs.

I don't know about that. If you scraped the same data and ran a search engine I think people would generally say you're fine. The copyright issue isn't the scraping step.

Even refined web runs about 2TB once loaded into Postgres with TS vector columns, and that’s a substantially smaller dataset than common crawl.

It’s not just a dumping a to of zip files on your NAS, it’s making the data responsive and usable.

YMMV depending on the value of "you" and your budget.

If you're Google, Amazon or even lower tier companies like Comcast, Yahoo or OpenAI, you can scape a massive amount of data (ignoring the "allowed" here, because TFA is about OpenAI disregarding robots.txt)

Meta is happily training their own models with this data, so it isn't going to waste.

However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.

And when other mediums have been saturated with AI? Books, music, radio, podcasts, movies -- what then? Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything?

I’m trying to work on monetization for my product now. The “personal Google” idea is really just an accidental byproduct of solving a much harder task. Not sure if people would pay for that alone.

That said… are content creators collectively (all media, film and books as well as web) a thin tail or a fat tail?

I could easily believe most of the actual culture comes from 10k-100k people today, even if there's, IDK, ten million YouTubers or something (I have a YouTube channel, something like 14 k views over 14 years, this isn't "culturally relevant" scale, and even if it had been most of those views are for algorithmically generated music from 2010 that's a literal Markov chain).

It looks less like an ouroboros and more like a bootstrapping problem.

This problem can't really be avoided once we begin using AI to write, understand, explain, and disseminate information for us. It'll be writing more than blogs and SEO pages.

How long before we start readily using AI to write academic journals and scientific papers? It's really only a matter of time, if it's not already happening.

From “known good” sources of knowledge, we can generate an infinite amount of content. We can add more “known good” knowledge to the model by generating content about that knowledge and training on it.

I agree there will be many issues keeping up with what “known good” is, but that’s always been an issue.

That's my entire point -- AI only generates content right now, but it will also be the source of content for training purposes soon. We need a "known good" human knowledge-base, otherwise generative AI will degenerate as AI generated content proliferates.

Crawling the web, like in the case of the OP, isn't going to work for much longer. And books, video, and music are next.

That is training on content.

The future will have models pre-trained on content and tuned on corpuses of knowledge. The knowledge it is trained on will be a selling point for the model.

Think of it this way - if you want to update the model so it knows the latest news, does it matter if the news was AI generated if it was generated from details of actual events?

“It’s ok bro, another model will fix, just please, one more ~layer~ ~agent~ model”

It’s all fun and games until you can’t reliably generate your base models anymore, because all your _base_ data is too polluted.

Let’s not forget MS has a $10bn stake in the current crop of LLM’s turning out to be as magic as they claim, so I’m sure they will do anything to ensure that happens.

My point is about the inevitable future when _those_ models start to struggle.

The phi approach doesn’t seem like breaking the ouroboros, it just feels like inserting another model/snake into the loop.

What do you think the NSA is storing in that datacenter in Utah? Power point presentations? All that data is going to be trained into large models. Every phone call you ever had and every email you ever wrote. They are likely pumping enormous money into it as we speak, probably with the help of OpenAI, Microsoft and friends.

A buffer with several-days-worth of the entire internet's traffic for post-hoc decryption/analysis/filtering on interesting bits. All that tapped backbone/undersea cable traffic has to be stored somewhere.

They would almost be failing in their purpose if they were not doing this.

On the other hand, this is an incredibly tough signal to noise problem. I am not sure we really understand what kind of scaling properties this would have as far as finding signals.

encrypted with an algorithm currently considered to be un-brute-forcible. If you presume we'll be able to decrypt today's encrypted transmissions in, say, 50-100 years, I'd record the encrypted transmission if I were the NSA.

But is it big enough to store 50 years worth of encrypted transmissions?

Far cheaper to simply have spies infiltrate the ~3 companies that hold the keys to 98% of internet traffic.

What makes you think this is true? Yes, it's likely that the internet will have more AI generated content than real content eventually (if it hasn't happened already), but why do you think AI companies won't realize this and adjust their training methods?

It's funny whenever people bring this up, they think AI companies are some mindless juggernauts who will simply train without caring about data quality at all and end up with worse models that they'll still for some reason release. Don't people realize that attention to data quality is the core differentiating feature that lead companies like OpenAI to their market dominance in the first place?

My thoughts - I teach myself all the time. Self reflection with a loss function can lead to better results. Why can't the LLMs do the same (I grasp that they may not be programmed that way currently)? Top engines already do it with chess, go, etc. They exceed human abilities without human gameplay. To me that seems like the obvious and perhaps only route to general intelligence.

We as humans can recognize botnets. Why wouldn't the LLM? Sort of in a hierarchal boost - learn the language, learn about bots and botnets (by reading things like this discussion), learn to identify them, learn that their content doesn't help the loss function much, etc. I mean sure, if the main input is "as a language model I cannot..." and that is treated as 'gospel' that would lead to a poor LLM, but i don't think that is the future. LLMs are interacting with humans - how many times do they have to re-ask a question - that should be part of the learning/loss function. How often do they copy the text into their clipboard (weak evidence that the reply was good)? do you see that text in the wild, showing it was used? If so, in what context "Witness this horrible output of chatGPT: " should result in lower scores and suppression of that kind of thing.

I dream of the day where I have a local LLM (ie individualized, I don't care where the hardware is) as a filter on my internet. Never see a botnet again, or a stack overflow q/a that is just "this has already been answered" (just show me where it was answered), rewrite things to fix grammar, etc. We already have that with automatic translation of languages in your browser, but now we have the tools for something more intelligent than that. That sort of thing. Of course there will be an arms race, but in one sense who cares. If a bot is entirely indistinguishable from a person, is that a difference that matters? I can think of scenerios where the answer is an emphatic YES, but overall it seems like a net improvement.

This is the type of stuff the news organisations should be publishing about "AI". Instead I keep reading or hearing people referring to training data with phrases like, "The sum of all human knowledge..." Quite shocking anyone would believe that.

When a worker gets a webpage for the honeypot, it crawls it, scrapes it, and finds X links on the page where X is greater than 1. Those links get put on the crawler queue. Because there’s more than 1 link per page, each worker on the honeypot will add more links to the queue than it removed.

Other sites will eventually leave the queue, because they have a finite number of pages so the crawlers eventually have nothing new to queue.

Not on the honeypot. It has a virtually infinite number of pages. Scraping a page will almost deterministically increase the size of the queue (1 page removed, a dozen added per scrape). Because other sites eventually leave the queue, the queue eventually becomes just the honeypot.

OpenAI is big enough this probably wasn’t their entire queue, but I wouldn’t be surprised if it was a whole digit percentage. The author said 1.8M requests; I don’t know the duration, but that’s equivalent to 20 QPS for an entire day. Not a crazy amount, but not insignificant. It’s within the QPS Googlebot would send to a fairly large site like LinkedIn.