Project import generated by Copybara.

GitOrigin-RevId: 08c2016a4df5aef571b464a4d4491f38c6b2af10
wongfei · Jun 3, 2021 · 8b57bf8 · 8b57bf8
1 parent ae05ad0
commit 8b57bf8
Show file tree

Hide file tree

Showing 118 changed files with 3,999 additions and 391 deletions.
diff --git a/.github/ISSUE_TEMPLATE/00-build-installation-issue.md b/.github/ISSUE_TEMPLATE/00-build-installation-issue.md
@@ -0,0 +1,21 @@
+<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
+
+**System information** (Please provide as much relevant information as possible)
+- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
+- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
+- Installed using virtualenv? pip? Conda? (if python):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- XCode and Tulsi versions (if iOS):
+- Android SDK and NDK versions (if android):
+- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
+- OpenCV version (if running on desktop):
+
+**Describe the problem**:
+
+
+**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
+
+**Complete Logs:**
+Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:
diff --git a/.github/ISSUE_TEMPLATE/10-solution-issue.md b/.github/ISSUE_TEMPLATE/10-solution-issue.md
@@ -0,0 +1,20 @@
+<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
+
+**System information** (Please provide as much relevant information as possible)
+- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
+- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- Solution (e.g. FaceMesh, Pose, Holistic):
+- Programming Language and version ( e.g. C++, Python, Java):
+
+**Describe the expected behavior:**
+
+**Standalone code you may have used to try to get what you need :**
+
+If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
+
+**Other info / Complete Logs :**
+Include any logs or source code that would be helpful to
+diagnose the problem. If including tracebacks, please include the full
+traceback. Large logs and files should be attached:
diff --git a/.github/ISSUE_TEMPLATE/20-documentation-issue.md b/.github/ISSUE_TEMPLATE/20-documentation-issue.md
@@ -0,0 +1,45 @@
+Thank you for submitting a MediaPipe documentation issue.
+The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
+## URL(s) with the issue:
+
+Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
+
+## Description of issue (what needs changing):
+
+Kinds of documentation problems:
+
+### Clear description
+
+For example, why should someone use this method? How is it useful?
+
+### Correct links
+
+Is the link to the source code correct?
+
+### Parameters defined
+Are all parameters defined and formatted correctly?
+
+### Returns defined
+
+Are return values defined?
+
+### Raises listed and defined
+
+Are the errors defined? For example,
+
+### Usage example
+
+Is there a usage example?
+
+See the API guide:
+on how to write testable usage examples.
+
+### Request visuals, if applicable
+
+Are there currently visuals? If not, will it clarify the content?
+
+### Submit a pull request?
+
+Are you planning to also submit a pull request to fix the issue? See the docs
+https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
+
diff --git a/.github/bot_config.yml b/.github/bot_config.yml
@@ -0,0 +1,18 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+# A list of assignees
+assignees:
+   - sgowroji
diff --git a/.github/stale.yml b/.github/stale.yml
@@ -0,0 +1,34 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 7
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 7
+# Only issues or pull requests with all of these labels are checked if stale. Defaults to `[]` (disabled)
+onlyLabels:
+ - stat:awaiting response
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you.
+# Comment to post when removing the stale label. Set to `false` to disable
+unmarkComment: false
+closeComment: >
+  Closing as stale. Please reopen if you'd like to work on this further.
diff --git a/README.md b/README.md
@@ -40,11 +40,12 @@ Hair Segmentation
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                        | ✅                                                         | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |

diff --git a/WORKSPACE b/WORKSPACE
@@ -71,8 +71,8 @@ http_archive(
 # Google Benchmark library.
 http_archive(
     name = "com_google_benchmark",
-    urls = ["https://github.com/google/benchmark/archive/master.zip"],
-    strip_prefix = "benchmark-master",
+    urls = ["https://github.com/google/benchmark/archive/main.zip"],
+    strip_prefix = "benchmark-main",
     build_file = "@//third_party:benchmark.BUILD",
 )
 
@@ -369,9 +369,9 @@ http_archive(
 )
 
 # Tensorflow repo should always go after the other external dependencies.
-# 2021-04-30
-_TENSORFLOW_GIT_COMMIT = "5bd3c57ef184543d22e34e36cff9d9bea608e06d"
-_TENSORFLOW_SHA256= "9a45862834221aafacf6fb275f92b3876bc89443cbecc51be93f13839a6609f0"
+# 2021-05-27
+_TENSORFLOW_GIT_COMMIT = "d6bfcdb0926173dbb7aa02ceba5aae6250b8aaa6"
+_TENSORFLOW_SHA256 = "ec40e1462239d8783d02f76a43412c8f80bac71ea20e41e1b7729b990aad6923"
 http_archive(
     name = "org_tensorflow",
     urls = [

diff --git a/build_desktop_examples.sh b/build_desktop_examples.sh
@@ -97,6 +97,7 @@ for app in ${apps}; do
       if [[ ${target_name} == "holistic_tracking" ||
             ${target_name} == "iris_tracking" ||
             ${target_name} == "pose_tracking" ||
+            ${target_name} == "selfie_segmentation" ||
             ${target_name} == "upper_body_pose_tracking" ]]; then
         graph_suffix="cpu"
       else

diff --git a/docs/framework_concepts/calculators.md b/docs/framework_concepts/calculators.md
@@ -248,12 +248,58 @@ absl::Status MyCalculator::Process() {
 }
 ```
 
+## Calculator options
+
+Calculators accept processing parameters through (1) input stream packets (2)
+input side packets, and (3) calculator options. Calculator options, if
+specified, appear as literal values in the `node_options` field of the
+`CalculatorGraphConfiguration.Node` message.
+
+```
+  node {
+    calculator: "TfLiteInferenceCalculator"
+    input_stream: "TENSORS:main_model_input"
+    output_stream: "TENSORS:main_model_output"
+    node_options: {
+      [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+        model_path: "mediapipe/models/active_speaker_detection/audio_visual_model.tflite"
+      }
+    }
+  }
+```
+
+The `node_options` field accepts the proto3 syntax.  Alternatively, calculator
+options can be specified in the `options` field using proto2 syntax.
+
+```
+  node: {
+    calculator: "IntervalFilterCalculator"
+    node_options: {
+      [type.googleapis.com/mediapipe.IntervalFilterCalculatorOptions] {
+        intervals {
+          start_us: 20000
+          end_us: 40000
+        }
+      }
+    }
+  }
+```
+
+Not all calculators accept calcuator options. In order to accept options, a
+calculator will normally define a new protobuf message type to represent its
+options, such as `IntervalFilterCalculatorOptions`. The calculator will then
+read that protobuf message in its `CalculatorBase::Open` method, and possibly
+also in the `CalculatorBase::GetContract` function or its
+`CalculatorBase::Process` method. Normally, the new protobuf message type will
+be defined as a protobuf schema using a ".proto" file and a
+`mediapipe_proto_library()` build rule.
+
 ## Example calculator
 
 This section discusses the implementation of `PacketClonerCalculator`, which
 does a relatively simple job, and is used in many calculator graphs.
-`PacketClonerCalculator` simply produces a copy of its most recent input
-packets on demand.
+`PacketClonerCalculator` simply produces a copy of its most recent input packets
+on demand.
 
 `PacketClonerCalculator` is useful when the timestamps of arriving data packets
 are not aligned perfectly. Suppose we have a room with a microphone, light
@@ -279,8 +325,8 @@ input streams:
     imageframe of video data representing video collected from camera in the
     room with timestamp.
 
-Below is the implementation of the `PacketClonerCalculator`.  You can see
-the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
+Below is the implementation of the `PacketClonerCalculator`. You can see the
+`GetContract()`, `Open()`, and `Process()` methods as well as the instance
 variable `current_` which holds the most recent input packets.
 
 ```c++
@@ -401,6 +447,6 @@ node {
 The diagram below shows how the `PacketClonerCalculator` defines its output
 packets (bottom) based on its series of input packets (top).
 
-| ![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png) |
-| :---------------------------------------------------------------------------: |
-| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
+![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png)  |
+:--------------------------------------------------------------------------: |
+*Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
diff --git a/docs/framework_concepts/framework_concepts.md b/docs/framework_concepts/framework_concepts.md
@@ -111,11 +111,11 @@ component known as an InputStreamHandler.
 
 See [Synchronization](synchronization.md) for more details.
 
-### Realtime data streams
+### Real-time streams
 
 MediaPipe calculator graphs are often used to process streams of video or audio
 frames for interactive applications. Normally, each Calculator runs as soon as
 all of its input packets for a given timestamp become available. Calculators
-used in realtime graphs need to define output timestamp bounds based on input
+used in real-time graphs need to define output timestamp bounds based on input
 timestamp bounds in order to allow downstream calculators to be scheduled
-promptly. See [Realtime data streams](realtime.md) for details.
+promptly. See [Real-time Streams](realtime_streams.md) for details.
diff --git a/docs/framework_concepts/realtime.md → docs/framework_concepts/realtime_streams.md b/docs/framework_concepts/realtime.md → docs/framework_concepts/realtime_streams.md
@@ -1,29 +1,28 @@
 ---
 layout: default
-title: Processing real-time data streams
+title: Real-time Streams
+parent: Framework Concepts
 nav_order: 6
-has_children: true
-has_toc: false
 ---
 
-# Processing real-time data streams
+# Real-time Streams
 {: .no_toc }
 
 1. TOC
 {:toc}
 ---
 
-## Realtime timestamps
+## Real-time timestamps
 
 MediaPipe calculator graphs are often used to process streams of video or audio
 frames for interactive applications. The MediaPipe framework requires only that
 successive packets be assigned monotonically increasing timestamps. By
-convention, realtime calculators and graphs use the recording time or the
+convention, real-time calculators and graphs use the recording time or the
 presentation time of each frame as its timestamp, with each timestamp indicating
 the microseconds since `Jan/1/1970:00:00:00`. This allows packets from various
 sources to be processed in a globally consistent sequence.
 
-## Realtime scheduling
+## Real-time scheduling
 
 Normally, each Calculator runs as soon as all of its input packets for a given
 timestamp become available. Normally, this happens when the calculator has
@@ -38,7 +37,7 @@ When a calculator does not produce any output packets for a given timestamp, it
 can instead output a "timestamp bound" indicating that no packet will be
 produced for that timestamp. This indication is necessary to allow downstream
 calculators to run at that timestamp, even though no packet has arrived for
-certain streams for that timestamp. This is especially important for realtime
+certain streams for that timestamp. This is especially important for real-time
 graphs in interactive applications, where it is crucial that each calculator
 begin processing as soon as possible.
 
@@ -83,12 +82,12 @@ For example, `Timestamp(1).NextAllowedInStream() == Timestamp(2)`.
 
 ## Propagating timestamp bounds
 
-Calculators that will be used in realtime graphs need to define output timestamp
-bounds based on input timestamp bounds in order to allow downstream calculators
-to be scheduled promptly. A common pattern is for calculators to output packets
-with the same timestamps as their input packets. In this case, simply outputting
-a packet on every call to `Calculator::Process` is sufficient to define output
-timestamp bounds.
+Calculators that will be used in real-time graphs need to define output
+timestamp bounds based on input timestamp bounds in order to allow downstream
+calculators to be scheduled promptly. A common pattern is for calculators to
+output packets with the same timestamps as their input packets. In this case,
+simply outputting a packet on every call to `Calculator::Process` is sufficient
+to define output timestamp bounds.
 
 However, calculators are not required to follow this common pattern for output
 timestamps, they are only required to choose monotonically increasing output