Text Gen SolutionFast,cost-optimizedLLM endpoints
Quickly evaluate and scale the latest models by leveraging OctoAI's singular API. Our deep expertise in model compilation, model curation, and ML systems means you get low-latency, affordable endpoints that can handle any production workload.
Read about our customers
Latitude Games lowers costs by 5x and unlocks new game experiences with Mixtral on OctoAI
Run your choice of models and fine-tunes
Build on your choice of OSS LLMs or your own model on our blazing fast API endpoints. Scale seamlessly and reliably without dropping performance.
Robust reliability
Serving millions of customer inferences daily
Adaptive scalability
Growth-ready for your app
Low cost with high performance
Keeping customers and the finance departments happy
Migrate with Ease
OpenAI SDK users move to OctoAI's compatible API with minimal effort
Stay up to date with new models and features
Product & Customer Updates
A Framework for Selecting the Right LLM
GitView launches AI code review analysis for engineering teams using OctoAI
30 Days of Llama 3: Newest Member of the Herd is Living up to the Hype
Fine-tuned Mistral 7B delivers over 60x lower costs and comparable quality to GPT 4
Latest Models
Hermes 2 Pro Llama 3
Llama 3 Instruct
Mixtral-8x22B Instruct
Mixtral 8x22B fine-tuned
Product & Customer Updates
A Framework for Selecting the Right LLM
GitView launches AI code review analysis for engineering teams using OctoAI
30 Days of Llama 3: Newest Member of the Herd is Living up to the Hype
Fine-tuned Mistral 7B delivers over 60x lower costs and comparable quality to GPT 4
Demos & Webinars
Selecting the right GenAI model for production
Watch our on-demand webinar as our engineers review all steps of model evaluation, testing, when to use checkpoints vs LoRAs, and how to get the best results.
PDF Summarizer
Learn how to build a PDF summary app using NodeJS and OctoAI’s Text Gen Solution. Convert PDFs into summarized TXT files with an LLM.
Recipe generator app with Llama 2 and OctoAI SDK
Learn how to build an app that takes a user list and outputs a recipe based on the input using Python and Llama 2 13B Chat.
Simple Chat
Learn how to build a ”hello world” question and answer chat app that uses Llama 2 70B, OctoAI, and LangChain. You just need a Python interpreter to get started.
Trusted by GenAI Innovators
“Working with the OctoAI team, we were able to quickly evaluate the new model, validate its performance through our proof of concept phase, and move the model to production. Mixtral on OctoAI serves a majority of the inferences and end player experiences on AI Dungeon today.”
Nick Walton
CEO & Co-Founder Latitude
“The LLM landscape is changing almost every day, and we need the flexibility to quickly select and test the latest options. OctoAI made it easy for us to evaluate a number of fine tuned model variants for our needs, identify the best one, and move it to production for our application.”
Matt Shumer
CEO & Co-Founder Otherside AI
JSON mode for reliable structured output
JSON mode is built into leading models on the OctoAI Systems Stack, allowing it to work without disruptions or quality issues. OctoAI has pushed further and optimized JSON mode for industry-leading latency performance.
Text embedding for RAG
Utilize GTE Large embedding endpoint to facilitate retrieval augmented generation (RAG) or semantic search for your apps. With a score of 63.13% on the MTEB leaderboard and compatible API, migrating from OpenAI requires minimal code updates. Learn how.
Build using our high quality and cost effective Mixtral 8x7B & 8x22B models
Our accelerated Mixtral delivers quality competitive with GPT 3.5, but with open source flexibility. Enjoy reduced costs with our 4x lower price per token than GPT 3.5. Migrating is made easy with one unified OpenAI compatible API. We support fine-tunes from the community including the latest from Nous Research.
MODEL COCKTAILSBuild using multiple models for your use case
Using OctoAI you can link several generative models together to create a highly performant pipeline. You can build new experiences specifically for your industry needs using language, images, audio, or your own custom models. Learn how our customer, Capitol AI, was able to work with us to achieve cost savings on their multiple models in production.