port code to tensorflow2.0

tf2.0/README.md

@@ -0,0 +1,47 @@
+## Edge Machine Learning: Tensorflow Library 
+
+This directory includes, Tensorflow implementations of various techniques and
+algorithms developed as part of EdgeML. Currently, the following algorithms are
+available in Tensorflow:
+
+1. [Bonsai](../docs/publications/Bonsai.pdf)
+2. [EMI-RNN](../docs/publications/emi-rnn-nips18.pdf)
+3. [FastRNN & FastGRNN](../docs/publications/FastGRNN.pdf)
+4. [ProtoNN](../docs/publications/ProtoNN.pdf)
+
+The TensorFlow compute graphs for these algoriths are packaged as
+`edgeml.graph`. Trainers for these algorithms are in `edgeml.trainer`. Usage
+directions and examples for these algorithms are provided in `examples`
+directory. To get started with any of the provided algorithms, please follow
+the notebooks in the the `examples` directory.
+
+## Installation
+
+Use pip and the provided requirements file to first install required
+dependencies before installing the `edgeml` library. Details for cpu based
+installation and gpu based installation provided below.
+
+It is highly recommended that EdgeML be installed in a virtual environment. Please create
+a new virtual environment using your environment manager ([virtualenv](https://virtualenv.pypa.io/en/stable/userguide/#usage) or [Anaconda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands)).
+Make sure the new environment is active before running the below mentioned commands.
+
+### CPU
+
+``` 
+pip install -r requirements-cpu.txt
+pip install -e .
+```
+
+Tested on Python3.5 and python 2.7 with >= Tensorflow 1.6.0.
+
+### GPU
+
+Install appropriate CUDA and cuDNN [Tested with >= CUDA 8.1 and cuDNN >= 6.1]
+
+```
+pip install -r requirements-gpu.txt
+pip install -e .
+```
+
+Copyright (c) Microsoft Corporation. All rights reserved.
+Licensed under the MIT license.

tf2.0/docs/FastCells.md

@@ -0,0 +1,57 @@
+# FastRNN and FastGRNN - FastCells
+
+This document aims to explain and elaborate on specific details of FastCells 
+present as part of `tf/edgeml/graph/rnn.py`. The endpoint use case scripts with 
+3 phase training along with an example notebook are present in `tf/examples/FastCells/`.
+One can use the endpoint script to test out the RNN architectures on any dataset 
+while specifying budget constraints as part of hyper-parameters in terms of sparsity and rank 
+of weight matrices.
+
+# FastRNN
+![FastRNN](img/FastRNN.png)
+![FastRNN Equation](img/FastRNN_eq.png)
+
+# FastGRNN
+![FastGRNN Base Architecture](img/FastGRNN.png)
+![FastGRNN Base Equation](img/FastGRNN_eq.png)
+
+# Plug and Play Cells
+
+`FastRNNCell` and `FastGRNNCell` present in `edgeml.graph.rnn` are very similar to 
+Tensorflow's inbuilt `RNNCell`, `GRUCell`, `BasicLSTMCell`, and `UGRNNCell` allowing us to 
+replace any of the standard RNN Cell in our architecture with FastCells. 
+One can see the plug and play nature at the endpoint script for FastCells, where the graph 
+building is very similar to LSTM/GRU in Tensorflow. 
+
+Script: [Endpoint Script](../examples/FastCells/fastcell_example.py)
+
+Example Notebook: [iPython Notebook](../examples/FastCells/fastcell_example.ipynb)
+
+Cells: [FastRNNCell](../edgeml/graph/rnn.py#L206) and [FastGRNNCell](../edgeml/graph/rnn.py#L31).
+
+# 3 phase Fast Training
+
+`FastCells`, similar to `Bonsai` use a 3 phase training routine, to induce the right 
+support and sparsity for the weight matrices. With the low-rank parameterization of weights 
+followed by the 3 phase training, we obtain FastRNN and FastGRNN models which are compact 
+and they can be further compressed by using byte quantization without significant loss in accuracy.
+
+# Compression
+
+1) Low-Rank Parameterization of Weight Matrices (L)
+2) Sparsity (S)
+3) Quantization (Q)
+
+Low-rank is directly induced into the FastCells during initialization and the training happens with 
+the targetted low-rank versions of the weight matrices. One can use `wRank` and `uRank` parameters 
+of FastCells to achieve this.
+
+Sparsity is taken in as hyper-parameter during the 3 phase training into `fastTrainer.py` which at the 
+end spits out a sparse, low-rank model.
+
+Further compression is achieved by byte Quantization and can be performed using `quantizeFastModels.py` 
+script which is part of `tf/exampled/FastCells/`. This will give model size reduction of up to 4x if 8-bit
+integers are used. Lastly, to facilitate all integer arithmetic, including the non-linearities, one could 
+use `quantTanh` instead of `tanh` and `quantSigm` instead of `sigmoid` as the non-linearities in the RNN 
+Cells followed by byte quantization. These non-linearities can be set using the appropriate parameters in 
+the `FastRNNCell` and `FastGRNNCell`

tf2.0/edgeml/__init__.py

@@ -0,0 +1,13 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT license.
+
+'''
+package edgeml
+
+Provides: Bonsai, ProtoNN and BasicTrainer routines
+    for both
+'''
+
+# TODO Override the __all__ variable for the package
+# and limit the functions that are exposed.
+# Do not expose functions in utils - can be dangerous

tf2.0/edgeml/graph/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT license.

tf2.0/edgeml/graph/bonsai.py

@@ -0,0 +1,180 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT license.
+
+import tensorflow as tf
+import numpy as np
+import warnings
+
+
+class Bonsai:
+    def __init__(self, numClasses, dataDimension, projectionDimension,
+                 treeDepth, sigma,
+                 isRegression=False, W=None, T=None, V=None, Z=None):
+        '''
+        Expected Dimensions:
+
+        Bonsai Params // Optional
+        W [numClasses*totalNodes, projectionDimension]
+        V [numClasses*totalNodes, projectionDimension]
+        Z [projectionDimension, dataDimension + 1]
+        T [internalNodes, projectionDimension]
+
+        internalNodes = 2**treeDepth - 1
+        totalNodes = 2*internalNodes + 1
+
+        sigma - tanh non-linearity
+        sigmaI - Indicator function for node probabilities
+        sigmaI - has to be set to infinity(1e9 for practicality)
+        while doing testing/inference
+        numClasses will be reset to 1 in binary case
+        '''
+        self.dataDimension = dataDimension
+        self.projectionDimension = projectionDimension
+        self.isRegression = isRegression
+
+        if ((self.isRegression == True) & (numClasses != 1)):
+            warnings.warn("Number of classes cannot be greater than 1 for regression")
+            self.numClasses = 1
+
+        if numClasses == 2:
+            self.numClasses = 1
+        else:
+            self.numClasses = numClasses
+
+        self.treeDepth = treeDepth
+        self.sigma = sigma
+
+        self.internalNodes = 2**self.treeDepth - 1
+        self.totalNodes = 2 * self.internalNodes + 1
+
+        self.W = self.initW(W)
+        self.V = self.initV(V)
+        self.T = self.initT(T)
+        self.Z = self.initZ(Z)
+
+        self.assertInit()
+
+        self.score = None
+        self.X_ = None
+        self.prediction = None
+
+    def initZ(self, Z):
+        if Z is None:
+            Z = tf.random.normal(
+                [self.projectionDimension, self.dataDimension])
+        Z = tf.Variable(Z, name='Z', dtype=tf.float32)
+        return Z
+
+    def initW(self, W):
+        if W is None:
+            W = tf.random.normal(
+                [self.numClasses * self.totalNodes, self.projectionDimension])
+        W = tf.Variable(W, name='W', dtype=tf.float32)
+        return W
+
+    def initV(self, V):
+        if V is None:
+            V = tf.random.normal(
+                [self.numClasses * self.totalNodes, self.projectionDimension])
+        V = tf.Variable(V, name='V', dtype=tf.float32)
+        return V
+
+    def initT(self, T):
+        if T is None:
+            T = tf.random.normal(
+                [self.internalNodes, self.projectionDimension])
+        T = tf.Variable(T, name='T', dtype=tf.float32)
+        return T
+
+    def __call__(self, X, sigmaI):
+        '''
+        Function to build the Bonsai Tree graph
+        Expected Dimensions
+
+        X is [_, self.dataDimension]
+        '''
+        errmsg = "Dimension Mismatch, X is [_, self.dataDimension]"
+        assert (len(X.shape) == 2 and int(
+            X.shape[1]) == self.dataDimension), errmsg
+        if self.score is not None:
+            return self.score, self.X_
+
+        X_ = tf.divide(tf.matmul(self.Z, X, transpose_b=True),
+                       self.projectionDimension)
+
+        W_ = self.W[0:(self.numClasses)]
+        V_ = self.V[0:(self.numClasses)]
+
+        self.__nodeProb = []
+        self.__nodeProb.append(1)
+
+        score_ = self.__nodeProb[0] * tf.multiply(
+            tf.matmul(W_, X_), tf.tanh(self.sigma * tf.matmul(V_, X_)))
+        for i in range(1, self.totalNodes):
+            W_ = self.W[i * self.numClasses:((i + 1) * self.numClasses)]
+            V_ = self.V[i * self.numClasses:((i + 1) * self.numClasses)]
+
+            T_ = tf.reshape(self.T[int(np.ceil(i / 2.0) - 1.0)],
+                            [-1, self.projectionDimension])
+            prob = (1 + ((-1)**(i + 1)) *
+                    tf.tanh(tf.multiply(sigmaI, tf.matmul(T_, X_))))
+
+            prob = tf.divide(prob, 2.0)
+            prob = self.__nodeProb[int(np.ceil(i / 2.0) - 1.0)] * prob
+            self.__nodeProb.append(prob)
+            score_ += self.__nodeProb[i] * tf.multiply(
+                tf.matmul(W_, X_), tf.tanh(self.sigma * tf.matmul(V_, X_)))
+
+        self.score = score_
+        self.X_ = X_
+        return self.score, self.X_
+
+    def getPrediction(self):
+        '''
+        Takes in a score tensor and outputs a integer class for each data point
+        '''
+
+        # Classification.
+        if (self.isRegression == False):
+            if self.prediction is not None:
+                return self.prediction
+
+            if self.numClasses > 2:
+                self.prediction = tf.argmax(input=tf.transpose(a=self.score), axis=1)
+            else:
+                self.prediction = tf.argmax(
+                    input=tf.concat([tf.transpose(a=self.score),
+                               0 * tf.transpose(a=self.score)], 1), axis=1)
+        # Regression.
+        elif (self.isRegression == True):
+            # For regression , scores are the actual predictions, just return them.
+            self.prediction = self.score
+
+        return self.prediction
+
+    def assertInit(self):
+        errmsg = "Number of Classes for regression can only be 1."
+        if (self.isRegression == True):
+            assert (self.numClasses == 1), errmsg
+        errRank = "All Parameters must has only two dimensions shape = [a, b]"
+        assert len(self.W.shape) == len(self.Z.shape), errRank
+        assert len(self.W.shape) == len(self.T.shape), errRank
+        assert len(self.W.shape) == 2, errRank
+        msg = "W and V should be of same Dimensions"
+        assert self.W.shape == self.V.shape, msg
+        errW = "W and V are [numClasses*totalNodes, projectionDimension]"
+        assert self.W.shape[0] == self.numClasses * self.totalNodes, errW
+        assert self.W.shape[1] == self.projectionDimension, errW
+        errZ = "Z is [projectionDimension, dataDimension]"
+        assert self.Z.shape[0] == self.projectionDimension, errZ
+        assert self.Z.shape[1] == self.dataDimension, errZ
+        errT = "T is [internalNodes, projectionDimension]"
+        assert self.T.shape[0] == self.internalNodes, errT
+        assert self.T.shape[1] == self.projectionDimension, errT
+        assert int(self.numClasses) > 0, "numClasses should be > 1"
+        msg = "# of features in data should be > 0"
+        assert int(self.dataDimension) > 0, msg
+        msg = "Projection should be  > 0 dims"
+        assert int(self.projectionDimension) > 0, msg
+        msg = "treeDepth should be >= 0"
+        assert int(self.treeDepth) >= 0, msg