Rust is a general-purpose language that can be used to create desktop applications and back-end server applications. It’s fully-compiled and feels similar to C++ when you’re programming with it. If you’re working for a company that relies heavily on Rust, you’ll likely need to know how to use the language to build and iterate on AI-based apps and services, especially considering industry interest in all things AI.
Let’s break down how Rust and AI intersect, as well as the best libraries and packages for quickly spinning up your Rust/AI experience.
Concurrency and AI
Because of Rust’s built-in concurrency, it’s easy to write code that will distribute itself across threads throughout the processor. (Most processors today have a mixture of cores that have two independent threads of execution and cores that have one thread.) So regardless of cores, it’s not uncommon these days to have 16 or more threads of execution in one CPU. That means your Rust app can distribute itself among up to 16 threads in such a case. (Remember you’ll be sharing the CPU with other apps, so you might not have quite that many available.)
Why is this important? Typically, AI apps need the thousands of cores available on a GPU; there are some that only need a dozen or so cores found on many desktop computers these days. That means you can use Rust to write some less resource-intensive AI apps.
Pro tip: If you’re exploring Rust concurrency for CPU-based AI, remember that, while CPU cores can have multiple threads each, with incorrect programming you might not actually be making use of them. The operating system (OS) manages its own threads, which are separate from CPU threads. The OS might pile multiple OS threads into a single CPU thread, and they each take turns running briefly. Each OS thread runs a bit, and then relinquishes control so the next thread can run. If you’re not careful, you might think you’re spinning up a dozen threads, when in fact you’re only using a single CPU thread, and your app will be incredibly slow.
If you’re interested in writing such parallel code for CPUs, look into the package called Rayon. It’s built for distributing your code across all the threads in the CPU’s cores. It fits nicely with the existing concurrency aspects of the Rust language.
Learning AI and Using Rust AI Packages
When you’re ready to start building AI apps, you’ll want to study up on how the different types of models work. Models are the foundation of today’s AI; they crunch through data and provide responses in the form of sentences, paragraphs, images, music, and so on depending on the type of model. For example, LLMs (which stands for Large Language Models) are for processing and generating human language.
Models have been trained with massive amounts of information (such as all of Wikipedia, as well as a huge number of websites). They learn from the information they’re given, and most today can generate information in return. L
If you want to use Rust to build an LLM, you have several potential avenues. You’ll want to explore the following Rust packages:
rust-bert: This a package that works with a type of LLM called Natural Language Processing (NLP). LLMs are a type of NLP.
tch-rs: This package lets you make use of a Python library called PyTorch. From there you can work directly with pretrained models.
gpt4all-rs: This package is Rust binding to a library written in C called llmodel. You can use it to connect with smaller models and run on the CPU.
You’ll probably want to learn a bit about how LLMs do their thing. Prompts are split up into individual words (sometimes partial words); those words are fed into the LLM, and each one gets a number called a token. The token depends on the context of the whole prompt; for example, the word “duck” in “I saw her duck” can differ based on context. If the context is a baseball game, it probably means “duck down.” If the context is a zoo, it might mean she’s holding the animal called a duck.
The LLM then uses probability and statistics to predict the first word of the response. Then it repeats the process to generate the next word, and the word after that, and so on, until it has a complete response.
Sentence Transformers in Rust
Sentence transformers are a special kind of model that operate similar to LLMs, except they process entire sentences as opposed to individual words or tokens. They return a vector of numbers, called an embedding, for each sentence. By encoding full sentences into a single vector, apps can quickly use cosines (from trigonometry) to calculate whether two sentences are similar in context and meaning.
Each cosine value is a decimal number between -1 and 1. If the cosine is close to 1, it means the two sentences are similar.
With this approach, entire documents can be processed and the vectors for each sentence stored in a database, along with the name of the document. A user can then “search” the documents by typing in a question or statement. The app will use the sentence transformer to calculate a vector for the sentence, and then search through the database to find the best match, the one with a cosine closest to 1.
If you’re interested in exploring this type of app in Rust, here are some libraries that will help you get started.
First, you’ll need to generate sentence embeddings. This starts by breaking up the sentences into individual tokens so they can be processed easily.
- tokenizers is a library best suited for this step.
Next you need to create embeddings (the vectors) for the sentences in the documents. For this you have a couple of choices:
- tch-rs, which we mentioned earlier, for loading models and generating embeddings
- Candle if you want to use the CPU cores instead of a GPU
After that, you need a vector database to store the embeddings. Some options include:
- Milvus: This is a vector database for storing the embeddings.
- Tantivy: This is written entirely in Rust and is extremely fast and optimized.
Next, you’ll need to calculate the cosine similarities. This doesn’t really require a library; instead, you can write code manually that does it. Take a look at this Wikipedia entry on calculating cosine similarity. The math is pretty straightforward.
Note: If you’re coming from the Python world, you might be familiar with the sentence transformers library, which is an all-in-one library for doing all of the above. Rust doesn’t have such a library; instead, it lets you choose the individual libraries for each step. There is, however, a library that does some of the above, called rust-bert. This library can do many of the steps but not the vector store or searching the vector store.
RAG Applications
While the sentence transformers can be used to search a set of documents and provide a knowledge base of sorts, you can take this to the next level by building a retrieval augmented generation (RAG) application by adding GPT and LLMs. Imagine that, instead of just returning the relevant document, you send the document along with the question to a chatbot like ChatGPT, and then have it read the document and provide an answer to your question based on what’s in the document. That’s the next step after sentence transformers. (Notice there are two models involved now: One does the sentence transformers, and the other, an LLM, does the generation.)
This step effectively combines the two sections we just covered, generating chat responses from an LLM along with the sentence transformers.
While you can accomplish this using the libraries already described, you have another option for the GPT step, and that’s to send the search results and original query to an online API-based GPT such as ChatGPT. To go that route, you need a good library for making HTTP API requests:
- reqwest: (Yes, spelled with a w!) This library is a generic library for making HTTP API requests.
But there are also a few community-built libraries that are specifically for interacting with OpenAI’s API:
Both of these provide the same functionality, for all intensive purposes. (You need to sign up at OpenAI and obtain a developer key.)
Machine Learning with Rust
With machine learning, an app continues to learn as it receives new data. The standard example here is how Amazon shows you product suggestions on their web page or app. As you purchase items, it learns more about your interests. (This is as opposed to, say, app stores that show a human-curated list of apps people might like.)
There are different flavors of machine learning, including supervised learning and unsupervised learning. With supervised learning, data and labels are fed in (such as a picture of an animal). As it analyzes the inputs, it learns how to classify them and apply the appropriate label. WIth unsupervised learning, no label is applied; the images are just analyzed and common patterns are found.
If you’re interested in exploring machine learning in the context of Rust, you can use these libraries:
- Linfa
- SmartCore: This one bills itself as the “most advanced machine learning library in Rust.” It provides several types of machine learning, including supervised and unsupervised learning.
Deep Learning
Deep learning is sort of like “next level” machine learning. Whereas machine learning uses mostly statistical analysis, deep learning relies on sophisticated neural networks to process incoming data and learn from it. An example here is image recognition.
The neural networks are trained with a large amount of data (such as hundreds of photos of individual species of animals); instead of using just statistical methods, the models start to find matching patterns of data between the images. When presented with an image it’s never seen before, it can match it to images it’s previously viewed and identify the type of animal.
Such systems can also learn human languages, what’s known as Natural Language Processing, the same way. For deep learning you can explore these libraries:
- Burn: This is a deep learning library written entirely in Rust.
- tch-rs and candle: We’ve already looked at these libraries earlier, and they can both handle deep learning.
Computer Vision
Using the tools I just described, models can be trained specifically for image recognition; from there, you can use libraries that work with these models. If you’re interested in exploring computer vision with Rust, you can check out this library:
- opencv-rust: This is a set of Rust bindings that allow Rust programmers to work with the OpenCV computer vision library.
Conclusion
This is just the tip of the iceberg, but it should be plenty to get you started building AI apps with Rust. Rust is a great language for AI because it’s compiled and therefore considerably faster than other languages such as Python and JavaScript. Add AI to your Rust skills and you’ll be ahead of the other job applicants, and soon you’ll land a Rust AI developer position.
