Even Open AI uses social media posts with WebText, a dataset which pulls text from outbound Reddit hyperlinks which acquired at the very least three karma, albeit with Wikipedia references taken off. "Every researcher producing a language model initial downloads Wikipedia then adds additional," Hagag stated.