Skip to content

Setup Guide

Jump to bottom
Pranav Prakash edited this page Jul 30, 2016 · 6 revisions

All the generic guidelines for setting up the octo a.i. environment. Very soon, we will be offering docker based setup so that most of the process is streamlined. But for now, this is how to get hands on.

Basic Setup

System

A standard *ix system with admin privilages. Tested and verified on

  • OS X 10.11.4 (El Capitan)
  • Ubuntu

Kong

Version 0.6.1

Kafka

Version: 0.8.2.2

We have found based on personal experience that other version of Kafka causes unstability. However, we are also aware that few developers have tried the latest kafka versions and it was working. You can try it on your own and update it here.

Cassandra

Version : 2.2.4

Redis

Version : 3.0.5

Zookeeper

Version : 3.4.7

Ruby

Preferred way: rvm
Version : 2.1.5

Repos

Before proceeding any further, make sure all the above mentioned systems are up and running. A good way to make sure everything is fine is to check their log files for anything suspicious.

Config first

  • Clone the config repo
  • The important files are config.yml and kong_config.yml.
  • Config.yml contains the configs for connecting to cassandra, kafka, zookeeper, redis, etc. It also defines AWS keys, logfile location, APNS gateway, mandrill keys, elasticsearch cluster details and so on. The values are typical for a local development setup and you should edit/verify them before going any further.
  • Kong_config.yml contains the config for your Kong setup. We use kong for API and microservices. The configs here would specify your web properties, their account details, rate limiting etc.

At this point, it is crucial to understand how different applications use this config. Each of the application has their own config directory. This directory contains a config.yml file which would contain configs very specific to that repo. All other common things are contained in this config repo. In order to make things work you would have to symlink this config directory to the config directory inside apps. If you do not understand this, you can still move forward. The steps are mentioned for each repo. Just make sure you have the correct configs here.

Gems Second

  • Clone the gems repo
  • Build and install all the gems by running the provided build.sh
  • Symlink the config to gems config by following
ln -fs /path/to/octo/config /path/to/octo/gems/octo-core/lib/octocore/config/config

NOTE:

  • Although we have pushed the gems to rubygems.org, but the database migration step is still dependent on running the rake task from the gems repo. So, this step becomes mandatory.

Creating Database

Once you have installed the octocore gem, you should be able to create the db schema depending on the environment. It is a rake task.

$ cd gems/octo-core/
$ rake --tasks
rake cequel:keyspace:create               # Initialize Cassandra keyspace
rake cequel:keyspace:create_if_not_exist  # Initialize Cassandra keyspace if not exist
rake cequel:keyspace:drop                 # Drop Cassandra keyspace
rake cequel:keyspace:drop_if_exist        # Drop Cassandra keyspace if exist
rake octo:init                            # Create keyspace and tables for all defined models
rake octo:migrate                         # Synchronize all models defined in `lib/octocore/models' with Cassandra database schema
rake octo:reset                           # Drop keyspace if exists, then create and migrate
rake spec                                 # Run RSpec code examples

$ rake octo:migrate
loading from file: lib/octocore/config/config.yml
loading from file: lib/octocore/config/search/index/user.yml
loading from file: lib/octocore/config/config/config.yml
loading from file: lib/octocore/config/config/kong_config.yml
loading from file: lib/octocore/config/search/index/user.yml
Synchronized schema for Octo::ContactUs
Synchronized schema for Octo::Enterprise
Synchronized schema for Octo::ApiEvent
Synchronized schema for Octo::ApiHit
Synchronized schema for Octo::ApiTrack
Synchronized schema for Octo::AppInit
Synchronized schema for Octo::AppLogin
Synchronized schema for Octo::AppLogout
Synchronized schema for Octo::Authorization
Synchronized schema for Octo::Category
Synchronized schema for Octo::CategoryBaseline
Synchronized schema for Octo::CategoryHit
Synchronized schema for Octo::CategoryTrend
Synchronized schema for Octo::Conversions
Synchronized schema for Octo::Ctr
Synchronized schema for Octo::DimensionChoice
Synchronized schema for Octo::EngagementTime
Synchronized schema for Octo::FunnelData
Synchronized schema for Octo::Funnel
Synchronized schema for Octo::GcmNotification
Synchronized schema for Octo::NewsfeedHit
Synchronized schema for Octo::NotificationHit
Synchronized schema for Octo::Page
Synchronized schema for Octo::PageView
Synchronized schema for Octo::PageloadTime
Synchronized schema for Octo::Product
Synchronized schema for Octo::ProductBaseline
Synchronized schema for Octo::ProductHit
Synchronized schema for Octo::ProductPageView
Synchronized schema for Octo::ProductTrend
Synchronized schema for Octo::PushKey
Synchronized schema for Octo::Rules
Synchronized schema for Octo::Segment
Synchronized schema for Octo::SegmentData
Synchronized schema for Octo::Tag
Synchronized schema for Octo::TagBaseline
Synchronized schema for Octo::TagHit
Synchronized schema for Octo::TagTrend
Synchronized schema for Octo::Template
Synchronized schema for Octo::Plan
Synchronized schema for Octo::Subscriber
Synchronized schema for Octo::User
Synchronized schema for Octo::PushToken
Synchronized schema for Octo::UserBrowserDetails
Synchronized schema for Octo::UserLocationHistory
Synchronized schema for Octo::UserPersona
Synchronized schema for Octo::UserPhoneDetails
Synchronized schema for Octo::UserProfileDetails
Synchronized schema for Octo::UserTimeline

NOTE:

  • Some ubuntu users have reported rake octo:migrate taking long time. In that case, please retry it a few times. It is known to work.

APIHandler

  • This repo handles all the API calls and pushes them to kafka
  • Clone APIHandler repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/apihandler/config/config
  • Use the provided bin/kong_setup.rb file in this repo alongwith the bin/kong_config.yaml to setup kong. Review the contents of yaml config file before.
  • Follow the instructions in provided README to setup APIHandlers.

APIConsumer

  • This repo handles all the kafka messages pushed by APIHandler and takes actions and triggers various hooks
  • Clone APIConsumer repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/apiconsumer/config/config
  • Follow the README instructions to get it working.

Recurring jobs

  • This repo is the hearbeat. It triggers various time based jobs.
  • Clone the Recurring Jobs repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/recurring-jobs/lib/config/config
  • Follow the instructions on provided README to start scheduler and workers.

Enterprise Dashboard

  • This is where business comes into picture. This is our end product that businesses use.
  • Clone the Enterprise Dashboard repo
  • Symlink config by
ln -s /path/to/octo/config /path/to/octo/enterprise-dashboard/config/config
  • As usual, follow the README

Newsfeed Webservice

Analytics Webservice

At this stage your development environment should be ready to serve the bare minimum of Octomatic platform.

You can use the fakestream command to create fake stream of data. Once you start fakestream, you would see a series of response tokens generated from the apihandler. This establishes the working proof of

  • API Key Setup
  • Kong setup
  • API Handler up and recieving

Further if you open redis-cli and look for keys. You should see a huge list of all kinds of keys that look like below. It is again a working proof of

  • API Consumer receiving messages
  • Event callbacks and triggers happening

Further you can also do cqlsh and see that the data is actually being written into the system.

To know more about this refer to https://github.com/octoai/gems/blob/master/octo-core/bin/fakestream

Actually placing it in your web app

  • Clone Javascript-client https://github.com/octoai/javascript-client
  • Follow its readme to create its production bundles
  • Copy those production bundles at a location like /assets/js/. If you have a different location, you may need to edit the below script accordingly.
  • Use the following Snippet in the HTML page where you want Octo-matic to track things for you.
<!-- Octo-matic Script -->
<script type="application/javascript">
  var OCTO_API_KEY = "YOUR_OCTO_API_KEY";
  var octo_opts = {
    cookie_name: '_oid'
    base_url: 'analytics.mydomain.com'
  };

function loadScript(t,e){var o=document.createElement("script")
o.type="text/javascript",o.readyState?o.onreadystatechange=function(){("loaded"==o.readyState||"complete"==o.readyState)&&(o.onreadystatechange=null,e())}:o.onload=function(){e()},o.src=t,document.body.appendChild(o)}"undefined"==typeof jQuery?loadScript("/assets/js/octo-minified-0.1.js",function(){jQuery.octo.apikey=OCTO_API_KEY,jQuery.octo.init(octo_opts)}):loadScript("/assets/js/octo-minified-no-jquery-0.1.js",function(){jQuery.octo.apikey=OCTO_API_KEY,jQuery.octo.init(octo_opts)})


</script>
<!-- Octo-matic ends -->
Clone this wiki locally