My guess—and I’m not privy to any inside information here—is that operations at places that are building neural-net-based models look something like this:
“Harvest” people are responsible for obtaining data and as much associated information as possible, stuff like where data was obtained from, timestamps, any identity information, etc (handle “X” on platform “Y”). If a comment or webpage or whatever gets revised, maybe they keep multiple copies. They’ll avoid things like following infinite loops on websites, deal with corrupt archives, all that. That probably goes into some giant database. Maybe some of these guys buy access, like Google did with Reddit. A lot are just going to write spiders, scraping bots, whatever. A search engine fundamentally has to have a spider slurp down data to create its index to work at all, so I’m sure that Google is going to have a bunch of expertise in this area, built up over many years. They’ll probably retain this data in pretty raw, unprocessed form.
“Unification” people are going to work on the above. They write code to convert the above into a standardized-across-all-platforms format. Maybe they convert stuff to unify timezones or time formats on comments. Maybe they strip off formatting stuff added by particular platforms. Markdown or HTML titles, or titles in Microsoft Word documents, for example, probably wind up represented in one way. Where possible, maybe usernames get attached, if data is from a forum.
“Scoring” people work on the unified dataset. They are going to be developing some system—and there might be a number of parallel projects—for generating a score of various sorts that might be a useful value for either a model or training information. With this as well, search engines have to do this; Google’s PageRank, for example, tries to assess the “value” of a page, determine what’s most-useful and reliable information, so that it can order results (and maybe blacklist some). A lot of what search engines do isn’t just getting you hits, but putting the most-valuable ones at the top. If you can develop heuristics to rank information based on reliability or political position or any number of other things, those are going to be useful numbers.
“Pruning” people use information, probably including the “scoring” information, to winnow down the size of the data, eliminate data where the cost of inclusion in the training set—it’s costly to train a neural net, and reducing the size of the training corpus reduces that cost—outweighs the benefits. They probably try to cut out garbage, irrelevant binary data, poisoned data, duplicate data, misinformation, whatever.
“Training” people are going to do the training and probably develop systems for rating the value of the above “scores” in producing desirable output. Maybe some “scoring” heuristic is really super-useful in producing accurate and useful output as defined by some benchmark that these guys set up. Like maybe, oh, one “score” is based on whether information was obtained from a .gov TLD, or another whether a text classifier says that the language is English and the Lix school year readability is above a certain level. Could be many of these. Some of what these guys find may go back to the “scoring” guys in trying to guess what heuristics should be further developed (“known time of domain age to time content was initially spidered is particularly valuable in identifying SEO-generated spam website content; see if you can produce more scores that identify mass-created spam websites”), some to the “pruning” guys to increase the low-value data that they prune out, and some of what they find may go into weighting importance of text for the training process. Some may even go all the way back to the “harvest” guys to determine how frequently to spider data from a given source—Google doesn’t respider information at a fixed rate, but hits “valuable” sources, like major news sites, very frequently. For non-pure-LLM neural nets—I don’t know if Google is doing this yet with Gemini, but I’d imagine that eventually, probably more-advanced AIs will need to move past just being an LLM—this might also be an input used in making a decision at runtime.
There are probably going to be a lot of points in this process where not-valuable data gets identified and removed or deprioritized. Even if no engineer at somewhere like Google has tried to write a “scoring” heuristic to identify text specifically to identify output from a tool that mass-edits comments on Reddit, my guess is that text consisting of nonsense words, which is what some of the Reddit mass-comment-editing-tools I’ve seen insert, probably intrinsically get identified as part of that process by multiple classification tools that generate heuristic scores. If all of a user’s comments look like the output of some simple text generation tool, or if all of them change in a given short period, that’s probably kind of a big red flag. That does mean that the information won’t be in the output model, but I think that people who are aiming to poison LLMs during training probably are expecting more impact from putting nonsense in their comment versus just blanking the comments than they’ll likely have. Google—and more broadly, people who are running search engines and even more, building neural nets trained on data on the Internet—are just going to have to already have a lot of fairly-sophisticated work in place to assess the value of and filter their inputs.
Since it’s scrambled and not deleted, hopefully future model updates will replace the original data.
But yes,it’s all a bit futile.
My guess—and I’m not privy to any inside information here—is that operations at places that are building neural-net-based models look something like this:
“Harvest” people are responsible for obtaining data and as much associated information as possible, stuff like where data was obtained from, timestamps, any identity information, etc (handle “X” on platform “Y”). If a comment or webpage or whatever gets revised, maybe they keep multiple copies. They’ll avoid things like following infinite loops on websites, deal with corrupt archives, all that. That probably goes into some giant database. Maybe some of these guys buy access, like Google did with Reddit. A lot are just going to write spiders, scraping bots, whatever. A search engine fundamentally has to have a spider slurp down data to create its index to work at all, so I’m sure that Google is going to have a bunch of expertise in this area, built up over many years. They’ll probably retain this data in pretty raw, unprocessed form.
“Unification” people are going to work on the above. They write code to convert the above into a standardized-across-all-platforms format. Maybe they convert stuff to unify timezones or time formats on comments. Maybe they strip off formatting stuff added by particular platforms. Markdown or HTML titles, or titles in Microsoft Word documents, for example, probably wind up represented in one way. Where possible, maybe usernames get attached, if data is from a forum.
“Scoring” people work on the unified dataset. They are going to be developing some system—and there might be a number of parallel projects—for generating a score of various sorts that might be a useful value for either a model or training information. With this as well, search engines have to do this; Google’s PageRank, for example, tries to assess the “value” of a page, determine what’s most-useful and reliable information, so that it can order results (and maybe blacklist some). A lot of what search engines do isn’t just getting you hits, but putting the most-valuable ones at the top. If you can develop heuristics to rank information based on reliability or political position or any number of other things, those are going to be useful numbers.
“Pruning” people use information, probably including the “scoring” information, to winnow down the size of the data, eliminate data where the cost of inclusion in the training set—it’s costly to train a neural net, and reducing the size of the training corpus reduces that cost—outweighs the benefits. They probably try to cut out garbage, irrelevant binary data, poisoned data, duplicate data, misinformation, whatever.
“Training” people are going to do the training and probably develop systems for rating the value of the above “scores” in producing desirable output. Maybe some “scoring” heuristic is really super-useful in producing accurate and useful output as defined by some benchmark that these guys set up. Like maybe, oh, one “score” is based on whether information was obtained from a .gov TLD, or another whether a text classifier says that the language is English and the Lix school year readability is above a certain level. Could be many of these. Some of what these guys find may go back to the “scoring” guys in trying to guess what heuristics should be further developed (“known time of domain age to time content was initially spidered is particularly valuable in identifying SEO-generated spam website content; see if you can produce more scores that identify mass-created spam websites”), some to the “pruning” guys to increase the low-value data that they prune out, and some of what they find may go into weighting importance of text for the training process. Some may even go all the way back to the “harvest” guys to determine how frequently to spider data from a given source—Google doesn’t respider information at a fixed rate, but hits “valuable” sources, like major news sites, very frequently. For non-pure-LLM neural nets—I don’t know if Google is doing this yet with Gemini, but I’d imagine that eventually, probably more-advanced AIs will need to move past just being an LLM—this might also be an input used in making a decision at runtime.
There are probably going to be a lot of points in this process where not-valuable data gets identified and removed or deprioritized. Even if no engineer at somewhere like Google has tried to write a “scoring” heuristic to identify text specifically to identify output from a tool that mass-edits comments on Reddit, my guess is that text consisting of nonsense words, which is what some of the Reddit mass-comment-editing-tools I’ve seen insert, probably intrinsically get identified as part of that process by multiple classification tools that generate heuristic scores. If all of a user’s comments look like the output of some simple text generation tool, or if all of them change in a given short period, that’s probably kind of a big red flag. That does mean that the information won’t be in the output model, but I think that people who are aiming to poison LLMs during training probably are expecting more impact from putting nonsense in their comment versus just blanking the comments than they’ll likely have. Google—and more broadly, people who are running search engines and even more, building neural nets trained on data on the Internet—are just going to have to already have a lot of fairly-sophisticated work in place to assess the value of and filter their inputs.
Well next time I’ll use a time machine and go back to 2007 to tell myself to stay on digg lol.
I guess the next step is to use a tool that poisons the old comments with incorrect information.