Pix2Pix

Pix2Pix

Image: Image of drawing converted to the Pokemon character, Pikachu using Pix2Pix trained on Pikachu images. Trained by Yining Shi;

Description

Image-to-image translation with conditional adversarial nets, or pix2pix, is a machine learning technique developed by Isola et al that learns how to map input images to output images.

The pix2pix model works by training on pairs of images such as building facade labels to building facades, and then attempts to generate the corresponding output image from any input image you give it. Source

The original pix2pix TensorFlow implementation was made by affinelayer. This version is heavily based on Christopher Hesse TensorFlow.js implementation

Quickstart

// Create a pix2pix model using a pre trained network
const pix2pix = ml5.pix2pix('models/customModel.pict', modelLoaded);
// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}
// Transfer using a canvas
pix2pix.transfer(canvas, (err, result) => {
  console.log(result);
});

Usage

Initialize

const styleTransfer = ml5.pix2pix(model, ?callback);

Parameters

model: REQUIRED. The path for a valid model.
callback: OPTIONAL. A function to run once the model has been loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.

Properties

.ready

Boolean to check if the model has loaded

Methods

.transfer()

Given an canvas element, applies image-to-image translation using the provided model. Returns an image.

styleTransfer.transfer(canvas, ?callback);

📥 Inputs

canvas: Required. A HTML canvas element.
callback: Optional. A function to run once the model has made the transfer. If no callback is provided, it will return a promise that will be resolved once the model has made the transfer.

📤 Outputs