FlagAI：面向大规模模型训练、微调与部署的开源工具包

FlagOpen; FlagAI

Repository navigation

Code; Pull requests; Actions; Projects; Security and quality; Insights

NotificationsYou must be signed in to change notification settings; Fork5 (5); Star18 (18)You must be signed in to star a repository

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.

Repository files navigation

简体中文

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model. Our goal is to support training, fine-tuning, and deployment of large-scale models on various downstream tasks with multi-modality.

Why should I use FlagAI?

Quickly Download Models via API FlagAI provides an API that allows you to quickly download pre-trained models and fine-tune them on a wide range of datasets collected from SuperGLUE and CLUE benchmarks for both Chinese and English text. FlagAI now supports over 30 mainstream models, including Language Model Aquila, multilingual text and image representation model AltCLIP, text-to-image generation model AltDiffusion , WuDao GLM (with a maximum of 10 billion parameters), EVA-CLIP, OPT, BERT, RoBERTa, GPT2, T5, ALM, and models from Huggingface Transformers, etc; Parallel train with fewer than 10 lines of code Backed by the four most popular data/model parallel libraries -- PyTorch, Deepspeed, Megatron-LM, BMTrain -- FlagAI allows for seamless integration between them, enabling users to parallel their training/testing process with fewer than ten lines of code; Conveniently use the few-shot learning toolkits FlagAI also provides prompt-learning toolkit for few-shot tasks; Particularly good at Chinese tasks These models can be applied to (Chinese/English) Text, for tasks like text classification, information extraction, question answering, summarization, and text generation, with a particular focus on Chinese tasks

Toolkits and Pre-trained Models

The code is partially based on GLM, Transformers，timm and DeepSpeedExamples.

Thanks for your interest in contributing! There are many ways to get involved; start with our contributor guidelines and then check these open issues for specific tasks.

Welcome to raise your questions or feature requests on GitHub Issues , and share your experience on the Discussions board.

Official email: open.platform@baai.ac.cn; Zhihu: FlagAI; Scan the qrcode to join the WeChat group for communication:

Quick Start

We provide many models which are trained to perform different tasks. You can load these models by AutoLoader to make prediction. See more in FlagAI/quickstart.

Requirements and Installation

Python version >= 3.8; PyTorch version >= 1.8.0; [Optional] For training/testing models on GPUs, you'll also need to install CUDA and NCCL

To install FlagAI with pip:

[Optional] To install FlagAI and develop locally:

[Optional] For faster training, install NVIDIA's apex

[Optional] For ZeRO optimizers, install DEEPSPEED (>= 0.7.7)

[Optional] For BMTrain training, install BMTrain (>= 0.2.2)

[Optional] For BMInf low-resource inference, install BMInf

[Optional] For Flash Attention, install Flash-attention (>=1.0.2)

[Tips] For single-node docker environments, we need to set up ports for your ssh. e.g., root@127.0.0.1 with port 711

[Tips] For multi-node docker environments, generate ssh keys and copy the public key to all nodes (in ~/.ssh/)

Load model and tokenizer

We provide the AutoLoad class to load the model and tokenizer quickly, for example:

This example is for the title_generation task, and you can also model other tasks by modifying the task_name. Then you can use the model and tokenizer to fine-tune or test.

Examples

1. Predictor

We provide the Predictor class to predict for different tasks, for example:

This example is for the seq2seq task, where we can get beam-search results by calling the predict_generate_beamsearch function. In addition, we also support prediction for tasks such as NER and title generate.

2. NER

3. Semantic Matching example

LICENSE

The majority of FlagAI is licensed under the Apache 2.0 license, however portions of the project are available under separate license terms:

Megatron-LM is licensed under the Megatron-LM license; GLM is licensed under the MIT license; AltDiffusion is licensed under the CreativeML Open RAIL-M license

[9 June 2023] release v1.7.0, Support Aquila #324; [31 Mar 2023] release v1.6.3, Support AltCLIP-m18 #303 and AltDiffusion-m18 #302; [17 Mar 2023] release v1.6.2, Support application of new optimizers #266, and added a new gpt model name 'GPT2-base-en' for English; [2 Mar 2023] release v1.6.1, Support Galactica model #234; BMInf, a low-resource inference package #238, and examples for p-tuning #227; [12 Jan 2023] release v1.6.0, support a new parallel lib called BMTrain and integate Flash Attention to speedup training of BERT and ViT models, examples in FlashAttentionBERT and FlashAttentionViT. Also add the contrastive search based text generation method SimCTG and DreamBooth finetuning based on AltDiffusion, examples in AltDiffusionNaruto; [28 Nov 2022] release v1.5.0, support 1.1B EVA-CLIP and [ALM: A large Arabic Language Model based on GLM], examples in ALM; [10 Nov 2022] release v1.4.0, support AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities, examples in AltCLIP and AltDiffusion; [29 Aug 2022] release v1.3.0, Added CLIP module and redesigned tokenizer APIs in #81; [21 Jul 2022] release v1.2.0, ViTs are supported in #71; [29 Jun 2022] release v1.1.0, support OPTs downloading and inference/fine-tuning #63; [17 May 2022] made our first contribution in #1

Platforms supported

Misc

↳ Stargazers, thank you for your support!

↳ Forkers, thank you for your support!

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.