TangiFlow

Table of Contents

• What is TangiFlow?
• Available Functionalities
• Installation
• Extending TangiFlow

What is TangiFlow?

TangiFlow is a tangible-based implementation of visual flow-based programming. It allows general-purpose programming via Python on a Microsoft Surface Hub. Users can use predefined nodes to specify data flow programs. TangiFlow is written in Swift and Python; Swift is responsible for front-end and Python is responsible for the backend of our system. To detect touch events we use MultiTouchKit (MTK), a Swift-based framework that handles touch events.

Toolbars

Shown below is the TangiFlow interface. The two toolbars on the sides of the screen contain all available functions:

Here is a toolbar zoomed in:

Nodes

Once the user selects an item from the toolbar, the corresponding functionality will appear as a node on workspace as shown below. Each node can have input and output fields and buttons to facilitate actions.

Source Nodes

Unlike other nodes, data source nodes do not take input from other nodes; instead, the folder button on the data source node can be used to select data source.

Branching the Output

Branch button increases amount of output arcs branching output of the node. New arcs will deliver the same data as other output arcs of the node.

Processing Nodes

Processing nodes as the name implies process the input data. They can have some extra input as textual data and those inputs can be provided either via keyboard or slider.

Available Functionalities

Users can use the following predefined functionalities to edit image files:

• Select image
• Change contrast
• Change brightness
• Change saturation
• Change sharpness
• Transform to black and white
• Apply filters (LoFi, Gingham, Inkwell, Valencia, and Kelvin)
• Stitch images (horizontal and vertical)
• Add caption

Users can also create their own functions using Python; see Extending Tangiflow

Installation

This git repository consists of two parts: Python back-end and Swift front-end. To run TangiFlow on a computer with macOS, a virtual machine that can run Windows, MTK, and a Microsoft Surface Hub are needed. Windows virtual machine runs an in-house software that sends touch events from the Surface Hub to the MTK. More information about MTK can be found in Rene Linden's thesis. FileHandler.swift file contains the paths to folders and files that TangiFlow uses.

var projectDataPath =  "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/Files/myproj.json"
var graphDataPath = "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/Files/graph.json"    
var resultFolderPath = "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/Files/Result"
var imagesFolderPath = "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/Files/Images"    
var mainScriptpath = "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/main.py"
var copyProj = "/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/Files/copy_proj.json"
var imageSource = "/Users/ppi/Desktop/ImageSource"

/Users/ppi/Documents/Code/MasterAsifPythonBackEnd/ is the path where our back-end is installed. You can change this part of paths to the location of python back-end on your machine. After installing the back-end and front-end, make sure to install dependencies. For the front-end you can use Carthage to install SwiftyJSON. This is the only dependency we are using (Carthage is dependency manager for Swift projects. You can check how to install dependencies from official repository). For the back-end you will need to install next libraries using PIP (PIP is a package installer for python. You can learn how to install packages using pip from official site):

• pythonflow
• cv2
• pilgram
• pil
• numpy
• textwrap

After all these dependencies are installed the software is ready to run on our setup. Simply open file MasterAsif2.workspace in XCode and run the project. Note: please make sure to checkout TangiFlow_Version branch from MTK repository and build that version of MTK. If you encounter error like one on the next screenshot just go to the path which stands after Referenced from and copy paste the MTK inside the Frameworks folder.

Extending TangiFlow

Developers can create their own functions. Currently, TangiFlow only supports generic Python data structures and images. Developers can add support for audio, video and other data types in future. All functions are stored in back-end location in the file Subroutines.py. A sample of this file:

import pythonflow as pf
from PythonToSwift.BackToSwift import BackToSwift
from PIL import ImageFilter,ImageEnhance,Image,ImageDraw,ImageFont
import numpy as np
import cv2 as cv
#
# next two routines are samples for image editing nodes
#
@pf.opmethod(length=1)
def image_source(controled_value,id):
    return Image.open(controled_value)


@pf.opmethod(length=1)
def change_contrast(image,controled_value,id):
    contrast = ImageEnhance.Contrast(image)
    result = contrast.enhance(float(controled_value))
    BackToSwift().prepare_data(type=ResultType.image.value,node_id=id, data=result,tool_type="pil",extension="jpg")
    return result
    
#
# next routine is sample for generic python data type routine
#
@pf.opmethod(length=1)
def do_something(data,id):
    #
    #do something with data
    #then assign it to result variable
    #
    result = ...
    BackToSwift().prepare_data(type=ResultType.generic_python.value,data=result)
    return result

Each function represent a node. There two types of nodes: source nodes and processing nodes. Source nodes provide data, processing nodes process provided data. Subroutines.py file must always import pythonflow and BackToSwift modules. After importing those modules you can import any other module you need in your project. Every function must have decorator @pf.opmethod(length=1). All processing functions must have argument which represents recieved data, in our sample this argument called image but you can name it as you want. If you want nodes in TangiFlow with controllable elements like text input, slider or file manager where you can pick file you need to add argument with controled_ prefix. As it can be seen source nodes does not have input data so they don't have argument which represents data. Using source nodes users can get data and source of the data is represented by controled_value argument. Later in this documentation I will explain how front-end detects and renders nodes based on arguments of functions. And at last all functions have id argument which uniquely represents each node on the screen in order to generate output file for each of them.

import pythonflow as pf
from PythonToSwift.BackToSwift import BackToSwift
@pf.opmethod(length=1)
def image_source(controled_value,id):
    return Image.open(controled_value)


@pf.opmethod(length=1)
def change_contrast(image,controled_value,id):
    contrast = ImageEnhance.Contrast(image)
    result = contrast.enhance(float(controled_value))
    BackToSwift().prepare_data(type=ResultType.image.value,node_id=id, data=result,tool_type="pil",extension="jpg")
    return result

After creating Subroutines.py file just run SourceFileParser.py file which will parse the routines file and will create a readable representation of functions for backend.

After this you also need to create two json files in back-end folder copy_proj.json and myproj.json with next content for image editing project:

{"config" : {
    "type" : "image",
    "tool_type" : "pil"
  }}

And with next content for generic python data types project:

{"config" : {
    "type" : "generic_python"
  }}

Locations of files was mentioned previously in this documentation.

In order to teach TangiFlow to use functions from Subroutines.py file as nodes we need to provide some data. Open myproj.json file. Nodes are represented as dictionary of json objects:

{
"config" : {
    "type" : "generic_python"
  },
  "PT-127.99.8" : {
    "info" : {
      "text" : "Lorem Ipsum Lorem Ipsum Lorem Ipsum",
      "img" : "\/Users\/ppi\/Documents\/Code\/MasterAsifPythonBackEnd\/Samples\/valencia.jpg"
    },
    "group_id" : "Apply filters",
    "alias" : "Apply Valencia filter",
    "arguments" : {
      "controled_args" : {

      },
      "main_args" : {
        "image" : "Image input"
      }
    },
    "function" : "apply_valencia"
  }
  
  }

This is sample of processing node PT-127.99.8 is unique id of this node. You can use any id you want as long as they dont match with other nodes' ids. Nodes with similar functionalities can be gathered in groups. In TangiFlow items with the same group_id will be grouped under item in toolbar with the title group_id. You also need to provide alias for node, alias is the name which will appear in front-end above every node. Function is the name of function from Subroutines.py which will handle action of respective node. Each function can have arguments. They have 2 types of arguments: main arguments and controled arguments. Main arguments represents input arcs of each node and they represent data input. Controled arguments are all other arguments which can be controled by user during interaction with TangiFlow. It can be text input, input with slider, selection of data (e.g. select image from file manager). As it was mentioned before all controled arguments must have controled_ prefix. Main arguments are quite simple they have key and value as a simple string. Key represents name of argument in Subroutines.py and value represents its alias in TangiFlow which will be displayed near input arc. Controled arguments on the other hand can be more complex. Sample of node where function gets input argument from slider on the node:

"PT-127.99.1" : {
    "alias" : "Change saturation",
    "arguments" : {
      "controled_args" : {
        "controled_value" : {
          "max" : "10",
          "step" : "0.1",
          "type" : "slider",
          "min" : "0",
          "default" : "0"
        }
      },
      "main_args" : {
        "image" : "Image input"
      }
    },
    "function" : "change_saturation",
    "group_id" : "Change image parameters"
  }

TangiFlow will interpret argument controled_value of function change_saturation from Subroutines.py as a slider with minimum value of 0 and maximum value of 10, with step of the slider 0.1 and with default value 0.

This the sample of node where argument of the function will be interpreted as button which opens file manager and user can pick image from file manager:

"PT-127.99.01" : {
    "alias" : "Load image",
    "arguments" : {
      "main_args" : {

      },
      "controled_args" : {
        "controled_value" : {
          "type" : "button",
          "alias" : "Load image"
        }
      }
    },
    "function" : "image_source"
  }

Once user chooses image from file manager path of this image is assigned to controled_value argument of image_source function in Subroutines.py file.

Next sample shows how users can create inputs for nodes:

"PT-127.99.11" : {
  "alias" : "Add caption to image",
  "arguments" : {
    "main_args" : {
      "image" : "Image input"
    },
    "controled_args" : {
      "controled_caption" : {
        "type" : "text",
        "alias" : "Type caption..."
      }
    }
  },
  "function" : "add_caption"
}

Users can specify two types of text fields: alphanumeric text field and just numeric. They can specify it in type section. In our sample it has type text but if users just want to use numeric input they canchange it to number. Info section in node's json structure is for toolbar. Nodes can be chosen from toolbars. Each button which represents a node can have info button which will give useful information for users in form of images. Users just can add path to the image which they want to show as hint for users and when user will press info button this image will appear as a hint.