Project import generated by Copybara.

GitOrigin-RevId: 9295f8ea2339edb71073695ed4fb3fded2f48c60
wongfei · Aug 13, 2020 · d7c287c · d7c287c
1 parent 6b0ab0e
commit d7c287c
Show file tree

Hide file tree

Showing 248 changed files with 10,351 additions and 5,323 deletions.
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1,6 @@
+global-exclude .git*
+global-exclude *_test.py
+
+recursive-include mediapipe/models *.tflite *.txt
+recursive-include mediapipe/modules *.tflite *.txt
+recursive-include mediapipe/graphs *.binarypb
diff --git a/README.md b/README.md
@@ -22,9 +22,9 @@ desktop/cloud, web and IoT devices.
 
 ## ML solutions in MediaPipe
 
-Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris                                                                                                      | Hands
-:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---:
-[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands)
+Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris 🆕                                                                                                    | Hands                                                                                                      | Pose 🆕
+:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :----:
+[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose)
 
 Hair Segmentation                                                                                                                       | Object Detection                                                                                                                     | Box Tracking                                                                                                                | Objectron                                                                                                             | KNIFT
 :-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
@@ -33,20 +33,21 @@ Hair Segmentation
 <!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
 <!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
 
-[]()                                                                          | Android | iOS | Desktop | Web | Coral
-:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
-[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)       | ✅       | ✅   | ✅       | ✅   | ✅
-[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                 | ✅       | ✅   | ✅       |     |
-[Iris](https://google.github.io/mediapipe/solutions/iris)                           | ✅       | ✅   | ✅       | ✅   |
-[Hands](https://google.github.io/mediapipe/solutions/hands)                         | ✅       | ✅   | ✅       | ✅   |
-[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅       |     | ✅       | ✅   |
-[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)   | ✅       | ✅   | ✅       |     | ✅
-[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)           | ✅       | ✅   | ✅       |     |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                 | ✅       |     |         |     |
-[KNIFT](https://google.github.io/mediapipe/solutions/knift)                         | ✅       |     |         |     |
-[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                   |         |     | ✅       |     |
-[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)        |         |     | ✅       |     |
-[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)               |         |     | ✅       |     |
+[]()                                                                          | Android | iOS | Desktop | Python | Web | Coral
+:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
+[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)       | ✅       | ✅   | ✅       |        | ✅   | ✅
+[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                 | ✅       | ✅   | ✅       |        |     |
+[Iris](https://google.github.io/mediapipe/solutions/iris) 🆕                         | ✅       | ✅   | ✅       |        | ✅   |
+[Hands](https://google.github.io/mediapipe/solutions/hands)                         | ✅       | ✅   | ✅       |        | ✅   |
+[Pose](https://google.github.io/mediapipe/solutions/pose) 🆕                         | ✅       | ✅   | ✅       | ✅      | ✅   |
+[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅       |     | ✅       |        | ✅   |
+[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)   | ✅       | ✅   | ✅       |        |     | ✅
+[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)           | ✅       | ✅   | ✅       |        |     |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                 | ✅       |     |         |        |     |
+[KNIFT](https://google.github.io/mediapipe/solutions/knift)                         | ✅       |     |         |        |     |
+[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                   |         |     | ✅       |        |     |
+[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)        |         |     | ✅       |        |     |
+[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)               |         |     | ✅       |        |     |
 
 ## MediaPipe on the Web
 
@@ -68,6 +69,7 @@ never leaves your device.
 *   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
 *   [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
 *   [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
+*   [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
 *   [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
 
 ## Getting started
@@ -86,8 +88,10 @@ run code search using
 
 ## Publications
 
-*   [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation from a Single
-    Image](https://mediapipe.page.link/iris-blog) in Google AI Blog
+*   [bazelPose - On-device Real-time Body Pose Tracking](https://mediapipe.page.link/bazelpose-blog)
+    in Google AI Blog
+*   [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
+    in Google AI Blog
 *   [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
     in Google Developers Blog
 *   [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)

diff --git a/WORKSPACE b/WORKSPACE
@@ -137,6 +137,25 @@ http_archive(
     urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
 )
 
+# 2020-07-09
+http_archive(
+    name = "pybind11_bazel",
+    strip_prefix = "pybind11_bazel-203508e14aab7309892a1c5f7dd05debda22d9a5",
+    urls = ["https://github.com/pybind/pybind11_bazel/archive/203508e14aab7309892a1c5f7dd05debda22d9a5.zip"],
+    sha256 = "75922da3a1bdb417d820398eb03d4e9bd067c4905a4246d35a44c01d62154d91",
+)
+
+http_archive(
+    name = "pybind11",
+    urls = [
+        "https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11/archive/v2.4.3.tar.gz",
+        "https://github.com/pybind/pybind11/archive/v2.4.3.tar.gz",
+    ],
+    sha256 = "1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d",
+    strip_prefix = "pybind11-2.4.3",
+    build_file = "@pybind11_bazel//:pybind11.BUILD",
+)
+
 http_archive(
     name = "ceres_solver",
     url = "https://github.com/ceres-solver/ceres-solver/archive/1.14.0.zip",

diff --git a/build_ios_examples.sh b/build_ios_examples.sh
@@ -58,6 +58,9 @@ apps="${app_dir}/*"
 for app in ${apps}; do
   if [[ -d "${app}" ]]; then
     target_name=${app##*/}
+    if [[ "${target_name}" == "common" ]]; then
+      continue
+    fi
     target="${app}:${target_name}"
 
     echo "=== Target: ${target}"

diff --git a/docs/getting_started/building_examples.md b/docs/getting_started/building_examples.md
@@ -422,3 +422,73 @@ Note: This currently works only on Linux, and please first follow
     This will open up your webcam as long as it is connected and on. Any errors
     is likely due to your webcam being not accessible, or GPU drivers not setup
     properly.
+
+## Python
+
+### Prerequisite
+
+1.  Make sure that Bazel and OpenCV are correctly installed and configured for
+    MediaPipe. Please see [Installation](./install.md) for how to setup Bazel
+    and OpenCV for MediaPipe on Linux and macOS.
+
+2.  Install the following dependencies.
+
+    ```bash
+    # Debian or Ubuntu
+    $ sudo apt install python3-dev
+    $ sudo apt install python3-venv
+    $ sudo apt install -y protobuf-compiler
+    ```
+
+    ```bash
+    # macOS
+    $ brew install protobuf
+    ```
+
+### Set up Python virtual environment.
+
+1.  Activate a Python virtual environment.
+
+    ```bash
+    $ python3 -m venv mp_env && source mp_env/bin/activate
+    ```
+
+2.  In the virtual environment, go to the MediaPipe repo directory.
+
+3.  Install the required Python packages.
+
+    ```bash
+    (mp_env)mediapipe$ pip3 install -r requirements.txt
+    ```
+
+4.  Generate and install MediaPipe package.
+
+    ```bash
+    (mp_env)mediapipe$ python3 setup.py gen_protos
+    (mp_env)mediapipe$ python3 setup.py install
+    ```
+
+### Run in Python interpreter
+
+Make sure you are not in the MediaPipe repo directory.
+
+Using [MediaPipe Pose](../solutions/pose.md) as an example:
+
+```bash
+(mp_env)$ python3
+>>> import mediapipe as mp
+>>> pose_tracker = mp.examples.UpperBodyPoseTracker()
+
+# For image input
+>>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
+>>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
+
+# For live camera input
+# (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
+>>> pose_tracker.run_live()
+
+# Close the tracker.
+>>> pose_tracker.close()
+```
+
+Tip: Use command `deactivate` to exit the Python virtual environment.
diff --git a/docs/getting_started/hello_world_ios.md b/docs/getting_started/hello_world_ios.md
@@ -18,8 +18,8 @@ This codelab uses MediaPipe on an iOS device.
 
 ### What you will learn
 
-How to develop an iOS application that uses MediaPipe and run a MediaPipe
-graph on iOS.
+How to develop an iOS application that uses MediaPipe and run a MediaPipe graph
+on iOS.
 
 ### What you will build
 
@@ -42,8 +42,8 @@ We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
 ```
 # MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
 # Used in the examples
-# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
-# mediapipe/examples/ios/edgedetectiongpu.
+# mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:helloworld
+# and mediapipe/examples/ios/helloworld.
 
 # Images coming into and out of the graph.
 input_stream: "input_video"
@@ -89,21 +89,21 @@ to build it.
 
 First, create an XCode project via File > New > Single View App.
 
-Set the product name to "EdgeDetectionGpu", and use an appropriate organization
+Set the product name to "HelloWorld", and use an appropriate organization
 identifier, such as `com.google.mediapipe`. The organization identifier
 alongwith the product name will be the `bundle_id` for the application, such as
-`com.google.mediapipe.EdgeDetectionGpu`.
+`com.google.mediapipe.HelloWorld`.
 
 Set the language to Objective-C.
 
 Save the project to an appropriate location. Let's call this
 `$PROJECT_TEMPLATE_LOC`. So your project will be in the
-`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu` directory. This directory will contain
-another directory named `EdgeDetectionGpu` and an `EdgeDetectionGpu.xcodeproj` file.
+`$PROJECT_TEMPLATE_LOC/HelloWorld` directory. This directory will contain
+another directory named `HelloWorld` and an `HelloWorld.xcodeproj` file.
 
-The `EdgeDetectionGpu.xcodeproj` will not be useful for this tutorial, as we will
-use bazel to build the iOS application. The content of the
-`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu/EdgeDetectionGpu` directory is listed below:
+The `HelloWorld.xcodeproj` will not be useful for this tutorial, as we will use
+bazel to build the iOS application. The content of the
+`$PROJECT_TEMPLATE_LOC/HelloWorld/HelloWorld` directory is listed below:
 
 1.  `AppDelegate.h` and `AppDelegate.m`
 2.  `ViewController.h` and `ViewController.m`
@@ -112,10 +112,10 @@ use bazel to build the iOS application. The content of the
 5.  `Main.storyboard` and `Launch.storyboard`
 6.  `Assets.xcassets` directory.
 
-Copy these files to a directory named `EdgeDetectionGpu` to a location that can
-access the MediaPipe source code. For example, the source code of the
-application that we will build in this tutorial is located in
-`mediapipe/examples/ios/EdgeDetectionGpu`. We will refer to this path as the
+Copy these files to a directory named `HelloWorld` to a location that can access
+the MediaPipe source code. For example, the source code of the application that
+we will build in this tutorial is located in
+`mediapipe/examples/ios/HelloWorld`. We will refer to this path as the
 `$APPLICATION_PATH` throughout the codelab.
 
 Note: MediaPipe provides Objective-C bindings for iOS. The edge detection
@@ -134,20 +134,20 @@ load(
 )
 
 ios_application(
-    name = "EdgeDetectionGpuApp",
-    bundle_id = "com.google.mediapipe.EdgeDetectionGpu",
+    name = "HelloWorldApp",
+    bundle_id = "com.google.mediapipe.HelloWorld",
     families = [
         "iphone",
         "ipad",
     ],
     infoplists = ["Info.plist"],
     minimum_os_version = MIN_IOS_VERSION,
     provisioning_profile = "//mediapipe/examples/ios:developer_provisioning_profile",
-    deps = [":EdgeDetectionGpuAppLibrary"],
+    deps = [":HelloWorldAppLibrary"],
 )
 
 objc_library(
-    name = "EdgeDetectionGpuAppLibrary",
+    name = "HelloWorldAppLibrary",
     srcs = [
         "AppDelegate.m",
         "ViewController.m",
@@ -172,31 +172,29 @@ The `objc_library` rule adds dependencies for the `AppDelegate` and
 `ViewController` classes, `main.m` and the application storyboards. The
 templated app depends only on the `UIKit` SDK.
 
-The `ios_application` rule uses the `EdgeDetectionGpuAppLibrary` Objective-C
-library generated to build an iOS application for installation on your iOS
-device.
+The `ios_application` rule uses the `HelloWorldAppLibrary` Objective-C library
+generated to build an iOS application for installation on your iOS device.
 
 Note: You need to point to your own iOS developer provisioning profile to be
 able to run the application on your iOS device.
 
 To build the app, use the following command in a terminal:
 
 ```
-bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
+bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:HelloWorldApp'
 ```
 
-For example, to build the `EdgeDetectionGpuApp` application in
-`mediapipe/examples/ios/edgedetectiongpu`, use the following
-command:
+For example, to build the `HelloWorldApp` application in
+`mediapipe/examples/ios/helloworld`, use the following command:
 
 ```
-bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
+bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/helloworld:HelloWorldApp
 ```
 
 Then, go back to XCode, open Window > Devices and Simulators, select your
 device, and add the `.ipa` file generated by the command above to your device.
-Here is the document on [setting up and compiling](./building_examples.md#ios) iOS
-MediaPipe apps.
+Here is the document on [setting up and compiling](./building_examples.md#ios)
+iOS MediaPipe apps.
 
 Open the application on your device. Since it is empty, it should display a
 blank white screen.
@@ -502,8 +500,8 @@ in our app:
 }];
 ```
 
-Note: It is important to start the graph before starting the camera, so that
-the graph is ready to process frames as soon as the camera starts sending them.
+Note: It is important to start the graph before starting the camera, so that the
+graph is ready to process frames as soon as the camera starts sending them.
 
 Earlier, when we received frames from the camera in the `processVideoFrame`
 function, we displayed them in the `_liveView` using the `_renderer`. Now, we
@@ -552,9 +550,12 @@ results of running the edge detection graph on a live video feed. Congrats!
 
 ![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif)
 
-If you ran into any issues, please see the full code of the tutorial
-[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
+Please note that the iOS examples now use a [common] template app. The code in
+this tutorial is used in the [common] template app. The [helloworld] app has the
+appropriate `BUILD` file dependencies for the edge detection graph.
 
 [Bazel]:https://bazel.build/
-[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
+[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/edge_detection/edge_detection_mobile_gpu.pbtxt
 [MediaPipe installation guide]:./install.md
+[common]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/common)
+[helloworld]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/helloworld)