Guardrails AI wants to provide fixes to GenAI model issues


It doesn’t take much for GenAI to spout falsehoods and untruths.

Last week we provided an example with chatbots from Microsoft and Google. declare a Super Bowl winner before the game even begins. The real problems begin, however, when GenAI hallucinations become harmful – endorsing – approve torture, to strenghten ethnic and racial stereotypes and write convincingly on conspiracy theories.

A growing number of suppliers, from incumbent operators like Nvidia And Selling power to startups like CalypsoAI, offer products that they claim can mitigate the unwanted and toxic content of GenAI. But these are black boxes; Unless you test each one independently, it’s impossible to know how these anti-hallucination products compare — and whether they actually deliver on their promises.

Shreya Rajpal saw this as a major problem and founded a company, Guardrail AIto try to resolve it.

“Most organizations… face the same set of issues around responsible deployment of AI applications and struggle to determine what the best and most efficient solution is,” Rajpal told TechCrunch in an interview by email. “They often end up reinventing the wheel in terms of managing all the risks they care about. »

According to Rajpal, surveys suggest that complexity – and by extension risk – is one of the main barriers preventing organizations from adopting GenAI.

A recent survey from Intel subsidiary found that compliance and privacy, reliability, high cost of implementation and lack of technical skills were concerns shared by about a quarter of companies implementing applications GenAI. In a separate investigation from Riskonnect, a risk management software provider, more than half of executives said they were concerned about employees making decisions based on inaccurate information from GenAI tools.

Rajpal, who previously worked at a self-driving startup and, after Apple acquisition of, within Apple’s Special Projects Group, co-founded Guardrails with Diego Oppenheimer, Safeer Mohiuddin and Zayd Simjee. Oppenheimer previously led Algorithmia, a machine learning operations platform, while Mohiuddin and Simjee held technology and engineering leadership roles at AWS.

In some ways, what Guardrails offers isn’t that different from what’s already on the market. The startup’s platform acts as a wrapper around GenAI models, particularly open source and proprietary text generation models (e.g. OpenAI’s GPT-4), to make these models seemingly more trustworthy, reliable and secure.

Guardrail AI

Image credits: Guardrail AI

But where Guardrails differs is its open source business model – the platform’s codebase is available on GitHub, free – and its participatory approach.

Through a marketplace called Guardrails Hub, Guardrails allows developers to submit modular components called “validators” that probe GenAI models for certain behavior, compliance, and performance metrics. Validators can be deployed, reused, and reused by other Guardrails developers and customers, serving as the basis for custom GenAI model moderation solutions.

“With the Hub, our goal is to create an open forum to share knowledge and find the most effective way to [further] Adoption of AI – but also to create a set of reusable guardrails that any organization can adopt,” Rajpal said.

Guardrails Hub validators range from simple rules-based checks to algorithms for detecting and mitigating issues in models. There are about 50 at present, ranging from hallucination detectors and policy violations to filters for proprietary information and insecure codes.

“Most companies have general, uniform checks for profanity, personally identifiable information, etc.,” Rajpal said. “However, there is no single, universal definition of what constitutes acceptable use for a specific organization and team. There are organization-specific risks that need to be monitored: for example, communication policies are different from one organization to another. With the Hub, we enable users to use the solutions we offer out-of-the-box, or use them to get a solid starter solution that they can further customize to their particular needs.

A hub for railing designs is an intriguing idea. But the skeptic in me wonders whether developers will bother contributing to a platform – and a nascent platform – without the promise of some form of compensation.

Rajpal is optimistic that they will, if only for the recognition – and by selflessly helping the industry build a “safer” GenAI.

“The Hub allows developers to see the types of risks other companies face and the safeguards they put in place to address and mitigate those risks,” she added. “Validators are an open source implementation of these guardrails that organizations can apply to their use cases.”

Guardrails AI, which does not yet charge for any services or software, recently raised $7.5 million in a funding round led by Zetta Venture Partners with participation from Factory, Pear VC, Bloomberg Beta, Github Fund and angles including a renowned AI expert. Ian Goodfellow. Rajpal says profits will be put toward expanding Guardrails’ six-person team and other open source projects.

“We talk to many people – companies, small startups and individual developers – who are failing to launch GenAI applications due to lack of necessary insurance and risk mitigation,” she continued. “This is a new problem that has not existed at this scale, due to the advent of ChatGPT and foundation models around the world. We want to be the ones to solve this problem.


Leave a Comment

Your email address will not be published. Required fields are marked *