BodyPix

BodyPix Header Image of Harriet Tubman

Description

As written by the developers of BodyPix:

“Bodypix is an open-source machine learning model which allows for person and body-part segmentation in the browser with TensorFlow.js. In computer vision, image segmentation refers to the technique of grouping pixels in an image into semantic areas typically to locate objects and boundaries. The BodyPix model is trained to do this for a person and twenty-four body parts (parts such as the left hand, front right lower leg, or back torso). In other words, BodyPix can classify the pixels of an image into two categories: 1) pixels that represent a person and 2) pixels that represent background. It can further classify pixels representing a person into any one of twenty-four body parts.”

Quickstart

  1. const bodypix = ml5.bodyPix(modelReady);
  3. function modelReady() {
  4.   // segment the image given
  5.   bodypix.segment(img, gotResults);
  6. }
  8. function gotResults(error, result) {
  9.   if (error) {
  10.     console.log(error);
  11.     return;
  12.   }
  13.   // log the result
  14.   console.log(result.backgroundMask);
  15. }

Usage

Initialize

  1. const bodyPix = new ml5.bodyPix(?video, ?options, ?callback);

Parameters

  • video: OPTIONAL. An HTMLVideoElement
  • callback: REQUIRED. A function to run once the model has been loaded.

  • options: OPTIONAL. An object to change the defaults (shown below). The available options are:

    1. {
    2.   multiplier: 0.75, // 1.0, 0.75, or 0.50, 0.25
    3.   outputStride: 16, // 8, 16, or 32, default is 16
    4.   segmentationThreshold: 0.5, // 0 - 1, defaults to 0.5
    5.   palette: {
    6.     leftFace: {
    7.       id: 0,
    8.       color: [110, 64, 170],
    9.     },
    10.     rightFace: {
    11.       id: 1,
    12.       color: [106, 72, 183],
    13.     },
    14.     rightUpperLegFront: {
    15.       id: 2,
    16.       color: [100, 81, 196],
    17.     },
    18.     rightLowerLegBack: {
    19.       id: 3,
    20.       color: [92, 91, 206],
    21.     },
    22.     rightUpperLegBack: {
    23.       id: 4,
    24.       color: [84, 101, 214],
    25.     },
    26.     leftLowerLegFront: {
    27.       id: 5,
    28.       color: [75, 113, 221],
    29.     },
    30.     leftUpperLegFront: {
    31.       id: 6,
    32.       color: [66, 125, 224],
    33.     },
    34.     leftUpperLegBack: {
    35.       id: 7,
    36.       color: [56, 138, 226],
    37.     },
    38.     leftLowerLegBack: {
    39.       id: 8,
    40.       color: [48, 150, 224],
    41.     },
    42.     rightFeet: {
    43.       id: 9,
    44.       color: [40, 163, 220],
    45.     },
    46.     rightLowerLegFront: {
    47.       id: 10,
    48.       color: [33, 176, 214],
    49.     },
    50.     leftFeet: {
    51.       id: 11,
    52.       color: [29, 188, 205],
    53.     },
    54.     torsoFront: {
    55.       id: 12,
    56.       color: [26, 199, 194],
    57.     },
    58.     torsoBack: {
    59.       id: 13,
    60.       color: [26, 210, 182],
    61.     },
    62.     rightUpperArmFront: {
    63.       id: 14,
    64.       color: [28, 219, 169],
    65.     },
    66.     rightUpperArmBack: {
    67.       id: 15,
    68.       color: [33, 227, 155],
    69.     },
    70.     rightLowerArmBack: {
    71.       id: 16,
    72.       color: [41, 234, 141],
    73.     },
    74.     leftLowerArmFront: {
    75.       id: 17,
    76.       color: [51, 240, 128],
    77.     },
    78.     leftUpperArmFront: {
    79.       id: 18,
    80.       color: [64, 243, 116],
    81.     },
    82.     leftUpperArmBack: {
    83.       id: 19,
    84.       color: [79, 246, 105],
    85.     },
    86.     leftLowerArmBack: {
    87.       id: 20,
    88.       color: [96, 247, 97],
    89.     },
    90.     rightHand: {
    91.       id: 21,
    92.       color: [115, 246, 91],
    93.     },
    94.     rightLowerArmFront: {
    95.       id: 22,
    96.       color: [134, 245, 88],
    97.     },
    98.     leftHand: {
    99.       id: 23,
    100.       color: [155, 243, 88],
    101.     },
    102.   },
    103. };

Properties

.video

Object. HTMLVideoElement if given in the constructor. Otherwise it is null.

.model

Object. The bodyPix model.

.modelReady

Boolean. Truthy value indicating the model has loaded.

.modelPath

String. The path to the model.

.config

Object. The configuration options of bodyPix.

Methods

.segment()

Allows you to segment a person from the background.

  1. bodyPix.segment(?input, ?options, callback);

📥 Inputs

  • input: HTMLImageElement | | HTMLVideoElement | ImageData | HTMLCanvasElement. NOTE: Videos can be added in the constructor.
  • options: Object. You can change the outputStride and segmentationThreshold
  • callback: Function. A function to handle the results of .segment(). Likely a function to do something with the segmented image.

📤 Outputs

  • Object: Returns an Object. The returns will either be a UInt8 array corresponding to the image array or a p5Image if p5.js is available.

    1. {
    2.   segmentation,
    3.   raw: {
    4.     personMask: null,
    5.     backgroundMask: null,
    6.   },
    7.   tensor: {
    8.     personMask: null,
    9.     backgroundMask: null,
    10.   },
    11.   personMask: null,
    12.   backgroundMask: null,
    13. }

.segmentWithParts()

Allows you to get the segmented body parts of the person.

  1. bodyPix.segmentWithParts(?input, ?options, callback);

📥 Inputs

  • input: HTMLImageElement | HTMLVideoElement | ImageData | HTMLCanvasElement. NOTE: Videos can be added in the constructor.
  • options: Object. You can change the outputStride and segmentationThreshold
  • callback: Function. A function to handle the results of .segment(). Likely a function to do something with the segmented image.

📤 Outputs

  • Object: Returns an Object. The returns will either be a UInt8 array corresponding to the image array or a p5Image if p5.js is available.

    1. {
    2.   segmentation: *ImageData*,
    3.   raw: {
    4.     personMask: *ImageData*,
    5.     backgroundMask: *ImageData*,
    6.     partMask: *ImageData*
    7.   },
    8.   tensor: {
    9.     personMask: *Tensor*,
    10.     backgroundMask: *Tensor*,
    11.     partMask: *Tensor*,
    12.   },
    13.   personMask: *P5Image*,
    14.   backgroundMask: *P5Image*,
    15.   partMask: *P5Image*,
    16.   bodyParts: *JSONObject*
    17. }

Examples

p5.js

p5 web editor

plain javascript

Demo

No demos yet - contribute one today!

Tutorials

No tutorials yet - contribute one today!

Acknowledgements

Contributors:

Credits:

Source Code