October 24, 2024 10:06 AM
Image Credit: VentureBeat via StableDiffusion
Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
The enterprise world is rapidly growing its usage of open source large language models (LLMs), driven by companies gaining more sophistication around AI – seeking greater control, customization, and cost efficiency.
While closed models like OpenAI’s GPT-4 dominated early adoption, open source models have since closed the gap in quality, and are growing at least as quickly in the enterprise, according to multiple VentureBeat interviews with enterprise leaders.
This is a change from earlier this year, when I reported that while the promise of open source was undeniable, it was seeing relatively slow adoption. But Meta’s openly available models have now been downloaded more than 400 million times, the company told VentureBeat, at a rate 10 times higher than last year, with usage doubling from May through July 2024. This surge in adoption reflects a convergence of factors – from technical parity to trust considerations – that are pushing advanced enterprises toward open alternatives.
“Open always wins,” declares Jonathan Ross, CEO of Groq, a provider of specialized AI processing infrastructure that has seen massive uptake of customers using open models. “And most people are really worried about vendor lock-in.”
Even AWS, which made a $4 billion investment in closed-source provider Anthropic – its largest investment ever – acknowledges the momentum. “We are definitely seeing increased traction over the last number of months on publicly available models,” says Baskar Sridharan, AWS’ VP of AI & Infrastructure, which offers access to as many models as possible, both open and closed source, via its Bedrock service.
The platform shift by big app companies accelerates adoption
It’s true that among startups or individual developers, closed-source models like OpenAI still lead. But in the enterprise, things are looking very different. Unfortunately, there is no third-party source that tracks the open versus closed LLM race for the enterprise, in part because it’s near impossible to do: The enterprise world is too distributed, and companies are too private for this information to be public. An API company, Kong, surveyed more than 700 users in July. But the respondents included smaller companies as well as enterprises, and so was biased toward OpenAI, which without question still leads among startups looking for simple options. (The report also included other AI services like Bedrock, which is not an LLM, but a service that offers multiple LLMs, including open source ones — so it mixes apples and oranges.)
Image from a report from the API company, Kong. Its July survey shows ChatGPT still winning, and open models Mistral, Llama and Cohere still behind.
But anecdotally, the evidence is piling up. For one, each of the major business application providers has moved aggressively recently to integrate open source LLMs, fundamentally changing how enterprises can deploy these models. Salesforce led the latest wave by introducing Agentforce last month, recognizing that its customer relationship management customers needed more flexible AI options. The platform enables companies to plug in any LLM within Salesforce applications, effectively making open source models as easy to use as closed ones. Salesforce-owned Slack quickly followed suit.
Oracle also last month expanded support for the latest Llama models across its enterprise suite, which includes the big enterprise apps of ERP, human resources, and supply chain. SAP, another business app giant, announced comprehensive open source LLM support through its Joule AI copilot, while ServiceNow enabled both open and closed LLM integration for workflow automation in areas like customer service and IT support.
“I think open models will ultimately win out,” says Oracle’s EVP of AI and Data Management Services, Greg Pavlik. The ability to modify models and experiment, especially in vertical domains, combined with favorable cost, is proving compelling for enterprise customers, he said.
A complex landscape of “open” models
While Meta’s Llama has emerged as a frontrunner, the open LLM ecosystem has evolved into a nuanced marketplace with different approaches to openness. For one, Meta’s Llama has more than 65,000 model derivatives in the market. Enterprise IT leaders must navigate these, and other options ranging from fully open weights and training data to hybrid models with commercial licensing.
Mistral AI, for example, has gained significant traction by offering high-performing models with flexible licensing terms that appeal to enterprises needing different levels of support and customization. Cohere has taken another approach, providing open model weights but requiring a license fee – a model that some enterprises prefer for its balance of transparency and commercial support.
This complexity in the open model landscape has become an advantage for sophisticated enterprises. Companies can choose models that match their specific requirements – whether that’s full control over model weights for heavy customization, or a supported open-weight model for faster deployment. The ability to inspect and modify these models provides a level of control impossible with fully closed alternatives, leaders say. Using open source models also often requires a more technically proficient team to fine-tune and manage the models effectively, another reason enterprise companies with more resources have an upper hand when using open source.
Meta’s rapid development of Llama exemplifies why enterprises are embracing the flexibility of open models. AT&T uses Llama-based models for customer service automation, DoorDash for helping answer questions from its software engineers, and Spotify for content recommendations. Goldman Sachs has deployed these models in heavily regulated financial services applications. Other Llama users include Niantic, Nomura, Shopify, Zoom, Accenture, Infosys, KPMG, Wells Fargo, IBM, and The Grammy Awards.
Meta has aggressively nurtured channel partners. All major cloud providers embrace Llama models now. “The amount of interest and deployments they’re starting to see for Llama with their enterprise customers has been skyrocketing,” reports Ragavan Srinivasan, VP of Product at Meta, “especially after Llama 3.1 and 3.2 have come out. The large 405B model in particular is seeing a lot of really strong traction because very sophisticated, mature enterprise customers see the value of being able to switch between multiple models.” He said customers can use a distillation service to create derivative models from Llama 405B, to be able to fine tune it based on their data. Distillation is the process of creating smaller, faster models while retaining core capabilities.
Indeed, Meta covers the landscape well with its other portfolio of models, including the Llama 90B model, which can be used as a workhorse for a majority of prompts, and 1B and 3B, which are small enough to be used on device. Today, Meta released “quantized” versions of those smaller models. Quantization is another process that makes a model smaller, allowing less power consumption and faster processing. What makes these latest special is that they were quantized during training, making them more efficient than other industry quantized knock-offs – four times faster at token generation than their originals, using a fourth of the power.
Technical capabilities drive sophisticated deployments
The technical gap between open and closed models has essentially disappeared, but each shows distinct strengths that sophisticated enterprises are learning to leverage strategically. This has led to a more nuanced deployment approach, where companies combine different models based on specific task requirements.
“The large, proprietary models are phenomenal at advanced reasoning and breaking down ambiguous tasks,” explains Salesforce EVP of AI, Jayesh Govindarajan. But for tasks that are light on reasoning and heavy on crafting language, for example drafting emails, creating campaign content, researching companies, “open source models are at par and some are better,” he said. Moreover, even the high reasoning tasks can be broken into sub-tasks, many of which end up becoming language tasks where open source excels, he said.
Intuit, the owner of accounting software Quickbooks, and tax software Turbotax, got started on its LLM journey a few years ago, making it a very early mover among Fortune 500 companies. Its implementation demonstrates a sophisticated approach. For customer-facing applications like transaction categorization in QuickBooks, the company found that its fine-tuned LLM built on Llama 3 demonstrated higher accuracy than closed alternatives. “What we find is that we can take some of these open source models and then actually trim them down and use them for domain-specific needs,” explains Ashok Srivastava, Intuit’s chief data officer. They “can be much smaller in size, much lower and latency and equal, if not greater, in accuracy.”
The banking sector illustrates the migration from closed to open LLMs. ANZ Bank, a bank that serves Australia and New Zealand, started out using OpenAI for rapid experimentation. But when it moved to deploy real applications, it dropped OpenAI in favor of fine-tuning its own Llama-based models, to accommodate its specific financial use cases, driven by needs for stability and data sovereignty. The bank published a blog about the experience, citing the flexibility provided by Llama’s multiple versions, flexible hosting, version control, and easier rollbacks. We know of another top-three U.S. bank that also recently moved away from OpenAI.
It’s examples like this, where companies want to leave OpenAI for open source, that have given rise to things like “switch kits” from companies like PostgresML that make it easy to exit OpenAI and embrace open source “in minutes.”
Infrastructure evolution removes deployment barriers
The path to deploying open source LLMs has been dramatically simplified. Meta’s Srinivasan outlines three key pathways that have emerged for enterprise adoption:
Cloud Partner Integration: Major cloud providers now offer streamlined deployment of open source models, with built-in security and scaling features.
Custom Stack Development: Companies with technical expertise can build their own infrastructure, either on-premises or in the cloud, maintaining complete control over their AI stack – and Meta is helping with its so-called Llama Stack.
API Access: For companies seeking simplicity, multiple providers now offer API access to open source models, making them as easy to use as closed alternatives. Groq, Fireworks, and Hugging Face are examples. All of them are able to provide you an inference API, a fine-tuning API, and basically anything that you would need or you would get from a proprietary provider.
Safety and control advantages emerge
The open source approach has also – unexpectedly – emerged as a leader in model safety and control, particularly for enterprises requiring strict oversight of their AI systems. “Meta has been incredibly careful on the safety part, because they’re making it public,” notes Groq’s Ross. “They actually are being much more careful about it. Whereas with the others, you don’t really see what’s going on and you’re not able to test it as easily.”
This emphasis on safety is reflected in Meta’s organizational structure. Its team focused on Llama’s safety and compliance is large relative to its engineering team, Ross said, citing conversations with the Meta a few months ago. (A Meta spokeswoman said the company does not comment on personnel information). The September release of Llama 3.2 introduced Llama Guard Vision, adding to safety tools released in July. These tools can:
Detect potentially problematic text and image inputs before they reach the model
Monitor and filter output responses for safety and compliance
Enterprise AI providers have built upon these foundational safety features. AWS’s Bedrock service, for example, allows companies to establish consistent safety guardrails across different models. “Once customers set those policies, they can choose to move from one publicly available model to another without actually having to rewrite the application,” explains AWS’ Sridharan. This standardization is crucial for enterprises managing multiple AI applications.
Databricks and Snowflake, the leading cloud data providers for enterprise, also vouch for Llama’s safety: “Llama models maintain the “highest standards of security and reliability,” said Hanlin Tang, CTO for Neural Networks
Intuit’s implementation shows how enterprises can layer additional safety measures. The company’s GenSRF (security, risk and fraud assessment) system, part of its “GenOS” operating system, monitors about 100 dimensions of trust and safety. “We have a committee that reviews LLMs and makes sure its standards are consistent with the company’s principles,” Intuit’s Srivastava explains. However, he said these reviews of open models are no different than the ones the company makes for closed-sourced models.
Data provenance solved through synthetic training
A key concern around LLMs is about the data they’ve been trained on. Lawsuits abound from publishers and other creators, charging LLM companies with copyright violation. Most LLM companies, open and closed, haven’t been fully transparent about where they get their data. Since much of it is from the open web, it can be highly biased, and contain personal information.
Many closed sourced companies have offered users “indemnification,” or protection against legal risks or claims lawsuits as a result of using their LLMs. Open source providers usually do not provide such indemnification. But lately this concern around data provenance seems to have declined somewhat. Models can be grounded and filtered with fine-tuning, and Meta and others have created more alignment and other safety measures to counteract the concern. Data provenance is still an issue for some enterprise companies, especially those in highly regulated industries, such as banking or healthcare. But some experts suggest these data provenance concerns may be resolved soon through synthetic training data.
“Imagine I could take public, proprietary data and modify them in some algorithmic ways to create synthetic data that represents the real world,” explains Salesforce’s Govindarajan. “Then I don’t really need access to all that sort of internet data… The data provenance issue just sort of disappears.”
Meta has embraced this trend, incorporating synthetic data training in Llama 3.2’s 1B and 3B models.
Regional patterns may reveal cost-driven adoption
The adoption of open source LLMs shows distinct regional and industry-specific patterns. “In North America, the closed source models are certainly getting more production use than the open source models,” observes Oracle’s Pavlik. “On the other hand, in Latin America, we’re seeing a big uptick in the Llama models for production scenarios. It’s almost inverted.”
What is driving these regional variations isn’t clear, but they may reflect different priorities around cost and infrastructure. Pavlik describes a scenario playing out globally: “Some enterprise user goes out, they start doing some prototypes…using GPT-4. They get their first bill, and they’re like, ‘Oh my god.’ It’s a lot more expensive than they expected. And then they start looking for alternatives.”
Market dynamics point toward commoditization
The economics of LLM deployment are shifting dramatically in favor of open models. “The price per token of generated LLM output has dropped 100x in the last year,” notes venture capitalist Marc Andreessen, who questioned whether profits might be elusive for closed-source model providers. This potential “race to the bottom” creates particular pressure on companies that have raised billions for closed-model development, while favoring organizations that can sustain open source development through their core businesses.
“We know that the cost of these models is going to go to zero,” says Intuit’s Srivastava, warning that companies “over-capitalizing in these models could soon suffer the consequences.” This dynamic particularly benefits Meta, which can offer free models while gaining value from their application across its platforms and products.
A good analogy for the LLM competition, Groq’s Ross says, is the operating system wars. “Linux is probably the best analogy that you can use for LLMs.” While Windows dominated consumer computing, it was open source Linux that came to dominate enterprise systems and industrial computing. Intuit’s Srivastava sees the same pattern: ‘We have seen time and again: open source operating systems versus non open source. We see what happened in the browser wars,” when open source Chromium browsers beat closed models.
Walter Sun, SAP’s global head of AI, agrees: “I think that in a tie, people can leverage open source large language models just as well as the closed source ones, that gives people more flexibility.” He continues: “If you have a specific need, a specific use case… the best way to do it would be with open source.”
Some observers like Groq’s Ross believe Meta may be in a position to commit $100 billion to training its Llama models, which would exceed the combined commitments of proprietary model providers, he said. Meta has an incentive to do this, he said, because it is one of the biggest beneficiaries of LLMs. It needs them to improve intelligence in its core business, by serving up AI to users on Instagram, Facebook, Whatsapp. Meta says its AI touches 185 million weekly active users, a scale matched by few others.
This suggests that open source LLMs won’t face the sustainability challenges that have plagued other open source initiatives. “Starting next year, we expect future Llama models to become the most advanced in the industry,” declared Meta CEO Mark Zuckerberg in his July letter of support for open source AI. “But even before that, Llama is already leading on openness, modifiability, and cost efficiency.”
Specialized models enrich the ecosystem
The open source LLM ecosystem is being further strengthened by the emergence of specialized industry solutions. IBM, for instance, has released its Granite models as fully open source, specifically trained for financial and legal applications. “The Granite models are our killer apps,” says Matt Candy, IBM’s global managing partner for generative AI. “These are the only models where there’s full explainability of the data sets that have gone into training and tuning. If you’re in a regulated industry, and are going to be putting your enterprise data together with that model, you want to be pretty sure what’s in there.”
IBM’s business benefits from open source, including from wrapping its Red Hat Enterprise Linux operating system into a hybrid cloud platform, that includes usage of the Granite models and its InstructLab, a way to fine-tune and enhance LLMs. The AI business is already kicking in. “Take a look at the ticker price,” says Candy. “All-time high.”
Trust increasingly favors open source
Trust is shifting toward models that enterprises can own and control. Ted Shelton, COO of Inflection AI, a company that offers enterprises access to licensed source code and full application stacks as an alternative to both closed and open source models, explains the fundamental challenge with closed models: “Whether it’s OpenAI, it’s Anthropic, it’s Gemini, it’s Microsoft, they are willing to provide a so-called private compute environment for their enterprise customers. However, that compute environment is still managed by employees of the model provider, and the customer does not have access to the model.” This is because the LLM owners want to protect proprietary elements like source code, model weights, and hyperparameter training details, which can’t be hidden from customers who would have direct access to the models. Since much of this code is written in Python, not a compiled language, it remains exposed.
This creates an untenable situation for enterprises serious about AI deployment. “As soon as you say ‘Okay, well, OpenAI’s employees are going to actually control and manage the model, and they have access to all the company’s data,’ it becomes a vector for data leakage,” Shelton notes. “Companies that are actually really concerned about data security are like ‘No, we’re not doing that. We’re going to actually run our own model. And the only option available is open source.’”
The path forward
While closed-source models maintain a market share lead for simpler use cases, sophisticated enterprises increasingly recognize that their future competitiveness depends on having more control over their AI infrastructure. As Salesforce’s Govindarajan observes: “Once you start to see value, and you start to scale that out to all your users, all your customers, then you start to ask some interesting questions. Are there efficiencies to be had? Are there cost efficiencies to be had? Are there speed efficiencies to be had?”
The answers to these questions are pushing enterprises toward open models, even if the transition isn’t always straightforward. “I do think that there are a whole bunch of companies that are going to work really hard to try to make open source work,” says Inflection AI’s Shelton, “because they got nothing else. You either give in and say a couple of large tech companies own generative AI, or you take the lifeline that Mark Zuckerberg threw you. And you’re like: ‘Okay, let’s run with this.’”
VB Daily
Stay in the know! Get the latest news in your inbox daily
By subscribing, you agree to VentureBeat’s Terms of Service.
Thanks for subscribing. Check out more VB newsletters here.
An error occured.
Information contained on this page is provided by an independent third-party content provider. This website makes no warranties or representations in connection therewith. If you are affiliated with this page and would like it removed please contact editor @payson.business
