The new AIGC open source library MMagic: Leading the Diffusion Model, beyond editing, and unleashing magic!

Hayo News

April 26th, 2023

View OriginalTranslated by Google

Since the birth of MMEditing, it has been the preferred algorithm library for many image super-resolution, editing and generation tasks. It has helped multiple research teams win more than 10 top international competitions and supported more than 100 GitHub ecological projects. After the iterative update of OpenMMLab 2.0 and the code merge with MMGeneration, MMEditing has become a powerful tool that supports the underlying vision algorithms based on GAN and traditional CNN.

Today, MMEditing will embrace Generative AI and officially change its name to MMagic ( Multimodal Advanced , G enerative, and Intelligent Creation), dedicated to building a more advanced and comprehensive AIGC open source algorithm library .

MMagic experience URL: https://github.com/open-mmlab/mmagic

Cooperating with MMEngine's excellent training and experiment management support, MMagic will provide researchers and AIGC enthusiasts with faster and more flexible experiment support, and help you explore AIGC.

In MMagic, we support tasks such as fine-tuning for Stable Diffusion, image editing, and image and video generation. In addition, we also support optimization strategies such as xFormers to improve training and inference efficiency.

For Diffusion Model , we provide the following "magic":

Support image generation based on Stable Diffusion and Disco Diffusion
Supports Finetune methods such as Dreambooth and DreamBooth LoRA
Supports ControlNet for controllable text-to-image generation
Support xFormers acceleration
Support video generation based on MultiFrame Render
Support calling the basic model and sampling strategy of Diffusers through Wrapper

In order to improve your "casting" efficiency , we have made the following adjustments to "Magic Circuit":

Support Pytorch 2.0 acceleration for 33 algorithms
Support for fast inference interfaces
Refactored MultiValLoop and MultiTestLoop, both support the evaluation of generation type indicators (eg FID) and reconstruction type indicators (eg SSIM), and support the evaluation of multiple data sets at one time
Refactor DataSample to support combination and split of batch dimensions
Refactor DataPreprocessor and unify the data format of various tasks during training and reasoning

fun play

Added Inferencer——

A few lines of code to quickly implement model inference

MMagic supports fast reasoning interface, you only need to specify the model, you can quickly call:

 plain text ANTLR4 Bash C C# css CoffeeScript CMake Dart Django Docker EJS Erlang Git Go GraphQL Groovy HTML Java JavaScript JSON JSX Kotlin LaTeX less Lua Makefile markdown MATLAB Markup Objective-C Perl PHP PowerShell .properties Protocol Buffers Python R Ruby Sass (Sass) Sass (Scss) Scheme SQL Shell Swift SVG TSX TypeScript WebAssembly YAML XML from mmagic.apis import MMagicInferencer# create an inferencer!magician = MMagicInferencer(model_name='stable_diffusion')text_prompts = 'A mecha robot in a favela in expressionist style'result_out_dir = 'robot.png'magician.infer(text=text_prompts, result_out_dir =result_out_dir)

You can also perform demo reasoning through the command line:

 plain text ANTLR4 Bash C C# css CoffeeScript CMake Dart Django Docker EJS Erlang Git Go GraphQL Groovy HTML Java JavaScript JSON JSX Kotlin LaTeX less Lua Makefile markdown MATLAB Markup Objective-C Perl PHP PowerShell .properties Protocol Buffers Python R Ruby Sass (Sass) Sass (Scss) Scheme SQL Shell Swift SVG TSX TypeScript WebAssembly YAML XML python demo/mmediting_inference_demo.py --model-name eg3d \ --model-config configs/eg3d/eg3d_cvt-official-rgb_afhq-512x512.py \ --model-ckpt https://download.openmmlab.com/mmediting/eg3d /eg3d_cvt-official-rgb_afhq-512x512-ca1dd7c9.pth \ --result-out-dir eg3d_output \ # Save pictures and videos `eg3d_output` --interpolation camera \ # Interpolate camera position --num-images 100 # During interpolation Generate 100 images

Support MultiLoop - one-time evaluation of multiple data sets

In order to facilitate users to evaluate multiple indicators of multiple data sets at one time, we provide MultiValLoop and MultiTestLoop

 plain text ANTLR4 Bash C C# css CoffeeScript CMake Dart Django Docker EJS Erlang Git Go GraphQL Groovy HTML Java JavaScript JSON JSX Kotlin LaTeX less Lua Makefile markdown MATLAB Markup Objective-C Perl PHP PowerShell .properties Protocol Buffers Python R Ruby Sass (Sass) Sass (Scss) Scheme SQL Shell Swift SVG TSX TypeScript WebAssembly YAML XML # 1. Use `MultiValLoop` instead of `ValLoop` provided by default in MMEngine val_cfg = dict(type='MultiValLoop')# 2. Configure evaluation indicators for different data sets div2k_evaluator = dict( type='EditEvaluator', metrics=dict (type='SSIM', crop_border=2, prefix='DIV2K'))set5_evaluator = dict( type='EditEvaluator', metrics=[dict(type='PSNR', crop_border=2, prefix='Set5'), dict(type='SSIM', crop_border=2, prefix='Set5'), ])val_evaluator = [div2k_evaluator, set5_evaluator]# 3. Configure dataset div2k_dataloader = dict(...)set5_dataloader = dict(...) val_dataloader = [div2k_dataloader, set5_dataloader]

MMagic supports multi-frame rendering via ControlNet

—— Cool image/long video generation

 plain text ANTLR4 Bash C C# css CoffeeScript CMake Dart Django Docker EJS Erlang Git Go GraphQL Groovy HTML Java JavaScript JSON JSX Kotlin LaTeX less Lua Makefile markdown MATLAB Markup Objective-C Perl PHP PowerShell .properties Protocol Buffers Python R Ruby Sass (Sass) Sass (Scss) Scheme SQL Shell Swift SVG TSX TypeScript WebAssembly YAML XML from mmagic.apis import MMagicInferencermagician = MMagicInferencer(model_name='controlnet_animation')prompt = 'aa handsome man, silver hair, smiling, play basketball' negative_prompt = 'longbody, lowres, bad anatomy, bad hands, missing fingers, ' + \ ' extra digit, fewer digits, cropped, worst quality, low quality'# download video from 'https://download.openmmlab.com/mmediting/data/basketball.mp4'video = 'basketball.mp4'save_path = '/path/ to/your/output/video.mp4'magician.infer(video=video, prompt=prompt, negative_prompt=negative_prompt, save_path=save_path)

Tired of research? Play some basketball!

All styles are supported!

SAM + MMagic = Generate Anything!

The popular SAM (segment anything model) can also provide more blessings for MMagic! If you want to make your own animation, you can move to OpenMMLab PlayGround:

https://github.com/open-mmlab/playground/blob/main/mmediting_sam/README.md