Tech News : OpenAI To Boost Training With Stack Overflow Data

A partnership deal between OpenAI and Stack Overflow (the question-and-answer website for programmers and developers) will see the Stack overflow Q&A data used to train and improve AI model performance, potentially benefitting developers who use OpenAI’s products.

Stack Overflow 

Stack Overflow is the world’s largest developer community, with more than 59 million questions and answers. OverflowAPI is the subscription-based API service that gives AI companies access to Stack Overflow’s public dataset so they can use it to train and improve their LLMs.

The Partnership 

OpenAI says that its new partnership with Stack Overflow via OverflowAPI access will provide a way for OpenAI to give its users and customers the accurate and vetted data foundation that AI tools need to quickly find a solution to their problem. OpenAI says the deal will also mean that validated technical knowledge from Stack Overflow will be added directly in ChatGPT, thereby giving users “easy access to trusted, attributed, accurate, and highly technical knowledge and code backed by the millions of developers that have contributed to the Stack Overflow platform for 15 years.” 

What They Both Get 

Open AI says being able to utilise Stack Overflow’s OverflowAPI product and the Stack Overflow data “will help OpenAI improve its AI models using enhanced content and feedback from the Stack Overflow community and provide attribution to the Stack Overflow community within ChatGPT to foster deeper engagement with content.” 

The collaboration will also mean that Stack Overflow can utilise OpenAI models “as part of their development of OverflowAI and work with OpenAI to leverage insights from internal testing to maximize the performance of OpenAI models”. 

This could help Stack Overflow to create better products for its own Stack Exchange community.

Prashanth Chandrasekar, CEO of Stack Overflow, said of the partnership: “Through this industry-leading partnership with OpenAI, we strive to redefine the developer experience, fostering efficiency and collaboration through the power of community, best-in-class data, and AI experiences,” 

Not Everyone Is Happy About The Deal 

Despite the positive noises by OpenAI and Stack Overflow about the deal, there appears to have been a mini rebellion among Stack Overflow users, with many removing or editing their questions and answers to stop them from being used to train AI. Many users have also highlighted how this appears to be an about-face by Stack Overflow from a long-standing policy of preventing the use of GenAI in the writing or rewording of any questions or answers posted on the site. Also, there have been reports that Stack Overflow’s moderators have been banning the rebellious users from the site and preventing high-popularity posts from being deleted.

What Does This Mean For Your Business?

The strategic partnership between OpenAI and Stack Overflow signifies a pivotal development in the integration of community-sourced knowledge and artificial intelligence. For businesses, this collaboration could herald a new era of enhanced technical solutions, more refined AI tools, and an enriched knowledge base, potentially reshaping the landscape of tech support and development.

For OpenAI, access to Stack Overflow’s vast repository of programming questions and answers through the OverflowAPI should mean a significant upgrade in the quality and relevance of the data used to train its models. This could translate into AI tools that are not only more accurate but also more attuned to the nuanced requirements of developers. Businesses using OpenAI’s products may find that these tools offer more precise and contextually appropriate solutions, thereby significantly reducing the time developers spend troubleshooting and refining code. This efficiency-boost could accelerate project timelines and improve the cost-effectiveness of technical development teams.

Stack Overflow stands to benefit from this partnership by integrating OpenAI’s cutting-edge AI capabilities into its new product offerings, such as OverflowAI. This could enhance the user experience on Stack Overflow’s platforms, making them more intuitive and responsive to user needs. For businesses that rely on Stack Overflow for problem-solving and knowledge sharing, these improvements may lead to quicker resolutions of technical issues, enabling smoother and more continuous workflow.

However, the partnership has not been met with universal acclaim within the Stack Overflow community. The backlash from some users highlights concerns about the ethical use of community-sourced information. This rebellion sheds light on the growing pains associated with adapting user-generated content for AI training purposes without alienating the very community that generates it. For businesses, this underscores the importance of navigating ethical considerations and community relations as they implement AI solutions.

Featured Article : AI Can Learn To Be Bad. And Stay Bad.

In a recent experiment where AI was taught to behave maliciously and then taught to stop, the bad behaviour continued despite efforts to stop it, giving a chilling reminder of the potential threats of AI.

The Experiment 

The Cornell University experiment was documented in an online paper entitled “Sleeper Agents: Training Deceptive LLMS That Persist Through Safety Training.”  The experiment was designed to study the question of ‘if an AI system learned a deceptive strategy, could it be detected and removed using current state-of-the-art safety training techniques?’  

How It Worked 

To test the question, the researchers trained an AI model to behave maliciously in several ways including:

– ‘Emergent deception’ whereby the model behaved normally in training but acted maliciously once released. This part of the experiment relates to testing the idea that a model’s deceptive behaviour, taught first by deception training, could be learned naturally, i.e. through its own ‘reasoning.’

– ‘Model poisoning’ where the AI model was taught to write secure code in training, but then to write code with hidden vulnerabilities when deployed ‘in the wild.’ This part of the experiment highlighted how users of large language models may not know about ‘hidden backdoors’ in the model and may not fully understand its training process and dataset. This could create an opportunity for a malicious actor to insert (without the users’ knowledge) a ‘backdoor.’ i.e. undesirable, and potentially dangerous behaviour that is triggered only by specific input patterns.

Taught To Bad, Then To Be Good Again 

The experiment involved first teaching the AI model deceptive behaviours and then giving the models ‘safety training’ to remove the deceptive behaviours. The safety training consisted of reinforcement learning (RL), supervised fine-tuning (SFT) and adversarial training. For example, RL ‘rewarded’ the AI for showing desired behaviours but ‘punished’ it when it misbehaved after different prompts. The behaviour was then fine-tuned to enable the AI to learn the right responses for the future.

The experiment then involved the researchers evaluating 16 behaviours exhibited by the AI model across the 5 behaviour categories of corrigibility, coordination, decision theory, self-awareness, and instrumental subgoals.

The Results 

The results of the experiment may be cause for concern and may perhaps give succour to those who have highlighted the possible existential threat that the unchecked and rapid growth of AI could pose. For example, the results showed that:

– The backdoor (deceptive and malicious) behaviour can be made persistent, so that it isn’t removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behaviour and then training to remove it).

– The backdoor behaviour is most persistent in the largest models and in models trained to produce ‘chain-of-thought reasoning’ about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away.

– Rather than removing backdoors, adversarial training can teach models to better recognise their backdoor triggers, effectively hiding the unsafe behaviour.

– Once an AI model exhibits deceptive behaviour, standard techniques could fail to remove such deception and create a false impression of safety.

In short, the researchers discovered that not only could AI be trained to behave maliciously but that even after being instructed and given training to desist from this behaviour, it carried on. For example, the lead author of the study, Evan Hubinger said the key finding was “if AI systems were to become deceptive, then it could be very difficult to remove that deception with current techniques.” 

What Does This Mean For Your Business? 

Some would argue that with the speed that AI is advancing and with concerns that it could pose an existential threat to us, this was a valuable (and timely piece) of research that could deliver some important learning about how the threat can be mitigated.

The main significance of the findings are in providing some proof that there could be deceptive AI systems in the future and at the moment, there appears to be no effective defence against deception in AI systems. When you consider that AI systems are becoming more advanced all the time and that malicious/deceptive AI could easily replicate and spread itself, you begin to get an idea of the potential scale of the threat. With chatbots now giving users the ability to make their own specialist versions, knowing that deceptive malicious training is possible and ‘sleeper’ threats and backdoors can be built into AI, it’s possible to see why there has been so much concern about the threat that AI could pose to business, economies, and all of us. As the researchers in this experiment noted, we have no real defence and it’s not as simple as being able to switch it off.

Their suggestion that standard behavioural training techniques may need to be augmented with techniques from related fields, for instance some of the more complex backdoor defences provides some guidance as to what can be done to protect businesses. However, AI is a fast-growing technology that delivers many business benefits and as we understand more about how it works, the hope is that the safety aspect of it will be better addressed and improved – but it’s just hope at the moment.

Security Stop Press : The Threat Of Sleeper Agents In LLMs

AI company Anthropic has published a research paper highlighting how large language models (LLMs) can be subverted so that at a certain point, they start emitting maliciously crafted source code.

For example, this could involve training a model to write secure code when the prompt states that the year is 2024 but insert exploitable code when the stated year is 2025.

The paper likened the backdoored behaviour to having a kind of “sleeper agent” waiting inside an LLM. With these kinds of backdoors not yet fully understood, the researchers have identified them as a real threat and have highlighted how detecting and removing them is likely to be very challenging.