MyMagic AI 批量推理集成 LlamaIndex：使用指南
Batch Inference With MyMagic AI: How-To Guide | LlamaIndex
MyMagic AI 的批量推理 API 已集成进 LlamaIndex，支持 Llama 70B、Mistral 7B、Mixtral 8x7B、CodeLlama70b 及 Embedding 模型，可处理数据提取、摘要、分类、情感分析、训练数据生成和嵌入等任务。用户将文件放入 AWS S3 或 GCS 的 session 文件夹后，通过 LlamaIndex 封装发起请求，可动态指定 prompt、模型和 max_tokens；官方建议尽量使用异步 acomplete 接口，目前不支持 chat/achat 方法。其训练数据生成速度相比 GPT-3.5 可提升 5-10 倍。
正文
当前显示原文，中文全文译文尚未就绪。
Follow us on
This is a guest post from MyMagic AI.
MyMagic AI allows processing and analyzing large datasets with AI. MyMagic AI offers a powerful API for batch inference (also known as offline or delayed inference) that brings various open-source Large Language Models (LLMs) such as Llama 70B, Mistral 7B, Mixtral 8x7B, CodeLlama70b, and advanced Embedding models to its users. Our framework is designed to perform data extraction, summarization, categorization, sentiment analysis, training data generation, and embedding, to name a few. And now it's integrated directly into LlamaIndex!
Explore our free and paid plans today.
Organize Your Data in an AWS S3 or GCS Bucket: Create a folder using your user ID assigned to you upon registration; Inside that folder, create another folder (called a "session") to store all the files you need for your tasks; Purpose of the 'Session' Folder: This "Session" folder keeps your files separate from others, making sure that your tasks run on the right set of files. You can name your session subfolder anything you like; Granting Access to MyMagic AI: To allow MyMagic AI to securely access your files in the cloud, follow the setup instructions provided in the MyMagic AI documentation
2. Install: Install both MyMagic AI’s API integration and LlamaIndex library:
sh
3. API Request: The llamaIndex library is a wrapper around MyMagic AI’s API. What it does under the hood is simple: it sends a POST request to the MyMagic AI API while specifying the model, storage provider, bucket name, session name, and other necessary details.
python
We have designed the integration to allow the user to set up the bucket and data together with the system prompt when instantiating the llm object. Other inputs, e.g. question (i.e. your prompt), model and max_tokens are dynamic requirements when submitting complete and acomplete requests.
python
This dynamic entry allows developers to experiment with different prompts and models in their workflow while also controlling for model output to cap their spending limit. MyMagic AI’s backend supports both synchronous requests (complete) and asynchronous requests (acomplete). It is advisable, however, to use our async endpoints as much as possible as batch jobs are inherently asynchronous with potentially long processing times (depending on the size of your data).
Currently, we do not support chat or achat methods as our API is not designed for real-time interactive experience. However, we are planning to add those methods in the future that will function in a “batch way”. The user queries will be aggregated and appended as one prompt (to give the chat context) and sent to all files at once.
Use Cases
While there are myriads of use cases, here we provide a few to help motivate our users. Feel free to embed our API in your workflows that are good fit for batch processing.
1. Extraction
Imagine needing to extract specific information from millions of files stored in a bucket. Information from all files will be extracted with one API call instead of a million sequential ones.
2. Classification
For businesses looking to classify customer reviews such as positive, neutral, and negative. With one request you can start processing the requests over the weekend and get them ready by Monday morning.
3. Embedding
Embedding text files for further machine learning applications is another powerful use case of MyMagic AI's API. You will be ready for your vector db in a matter of days not weeks.
4. Training (Fine-tuning) Data Generation
Imagine generating thousands of synthetic data for your fine-tuning tasks. With MyMagic AI’s API, you can reduce the generation time by a factor of 5-10x compared to GPT-3.5.
5. Transcription
MyMagic AI’s API supports different types of files, so it is also easy to batch transcribe many mp3 or mp4 files in your bucket.
The output from batch inference processes, often voluminous, can seamlessly integrate into LlamaIndex's RAG pipeline for effective data storage and retrieval.
This section demonstrates how to use the Llama3 model from the Ollama library coupled with BGE embedding to manage information storage and execute queries. Please ensure the following prerequisites are installed and Llama3 model is pulled:
sh
For this demo, we have run a batch summarization job on 5 Amazon reviews (but this might be millions in some real scenarios) and saved the results as reviews_1_5.json:
text
Now let’s embed and store this document and ask questions using LlamaIndex’s query engine. Bring in our dependencies:
python
Configure the embedding model and Llama3 model
python
Update settings for the indexing pipeline:
python
Now create our index, our query engine and run a query:
Now we know that the review 1 is the least favorite movie among these reviews.
Next Steps