Getting started with ElasticSearch on Node

“Searching,
Seek and Destroy
Searching,
Seek and Destroy” – Metallica

I recently had to set up ElasticSearch within a Node project.  I found the supporting documentation to be scattered and had a difficult time finding examples of what I would consider everyday production configurations.  So I’ve gathered a lot of what I learned here and hopefully this saves some of you a bit of time.

Get the elasticsearch-js client

First, use the elasticsearch-js npm package.  This seems to be actively developed.  Specifically, do not use the elasticsearchclient package as that seems to be end-of-lifed.  Create your client like:

ElasticSearch = require("elasticsearch")
client = new ElasticSearch.Client {host: 'localhost:9200'}

Create the index

Next, you’ll want to create your indices.  This is where it can get overwhelming.  ElasticSearch is great because it can be configured in countless ways, but understanding all the variants can be hard at the beginning.  Here is what I think would be a great place to start.  Using your ES client, create an index ‘users’:

client.indices.create
  index: "users"
   type: "user"
   body:
     settings:
       analysis:
        filter:
          mynGram:
             type: "nGram"
             min_gram: 2
             max_gram: 12
          analyzer:
             default_index:
               type: "custom"
               tokenizer: "standard"
               filter: [
                 "lowercase"
              ]
             ngram_indexer:
               type: "custom"
               tokenizer: "standard"
               filter: [
                "lowercase"
                "mynGram"
              ]
             default_search:
              type: "custom"
              tokenizer: "lowercase"
              filter: [
                "standard"
              ]
     mappings:
       dynamic: "true"
       properties:
         id:
           type: "string"
           index: "not_analyzed"
         name:
          type: "string"
          index_analyzer: "ngram_indexer"
         suggest:
           type: "completion"
           index_analyzer: "simple"
           search_analyzer: "simple"
           payloads : true

Here we have:

  • defined the default index and search analyzers (default_index, default_search).  These will be used to index all fields unless otherwise specified, such as in…
  • mappings.properties.name.  Here we want to use an ngram tokenizer on the user’s name so we can do fun things like search within names so searching on ‘ike’ would return Mike and Ike.
  • defined a completion suggester index in mappings.properties.suggest

Test the setup

That’s the bare bones getting started.  To test out what your indexer is doing, you can simply run a query, specify the indexer you want to test out, and you’ll get a list of all the tokens that are being generated from that indexer.

localhost:9200/users/_analyze?pretty&analyzer=ngram_indexer&text=Michael
Advertisements

Popping modals all night in iOS

Poppin bottles in the ice, like a blizzard
When we drink we do it right, gettin’ slizard
Sippin sizzurp in my ride, like Three 6
Now I’m feelin so fly like a G6
Like a G6, Like a G6
Now I’m feelin so fly like a G6

I recently discovered a personal knowledge gap in using Scroll Views, Navigation Controllers and modal windows to cleanly work together. Here’s a brief description:

Picture a deck of cards spread out in one long row of 52 cards, edged right up next to each other.  Picture your phone being able to show only one card at a time.  The user uses a modal view to select the next card to display.  When the card is selected, the modal is dismissed and that card is displayed.

Brief recap:

  • 9 hearts is displayed
  • you use modal view to select 3 spades
  • 3 spades is displayed

(Really brief ScrollView primer)  In iOS land, the phone screen is the ScrollView’s current frame, and the position of the cards are matched with the ScrollView’s current offset.  Imagine the ScrollView’s frame glides, or scrolls, across all 52 cards in either direction in an instant.

The problem I discovered happened when I was dismissing a modal view.  The presenter of the modal dismisses it and then messages its parent to scroll away from it and display the 3 spades.  What would happen is the scroll would display that section but those pages or ‘cards’ had not yet been built.  The offset was being set before the content was created.  For whatever reason, the ViewWillAppear method did not seem to be called for the newly selected card views.

This was difficult to debug because it wasn’t a simple error message I could search for, but more required a deeper understanding of the platform.  And it turns out the solution is quite elegant.

The dismissViewControllerAnimated call has a completion block parameter.  Perfect.  Put the messaging to the parent after we have dismissed the modal.

- (void)selectCard:(NSString *)cardNumber {
    [self dismissViewControllerAnimated:YES completion:^{
        [[self delegate] displayCard:cardNumber];
    }];
}

Worked like a charm.

iOS – Updating application state from a UITableViewCell

One of my current clients is a new restaurant where they want an iOS app from which their customers can order their food.  One of the requirements is allowing the user to select 0-n number of dishes for multiple dishes.  For example:

Side dishes

So the question is as the user is incrementing the values, where do we store this state?  The first thought is to just keep it in the UITableViewCell but this has major drawbacks, including:

  • the data is lost once the UITableViewCell is scrolled off screen, as it is re-used to display other dishes.
  • maintaining state in the view is just not clean design.  This is controller territory.

Protocols and Delegation

The sensible place to keep the data is in the controller.  Then the question becomes, “How do we call back to the controller from the UITableViewCell?”  

Using Protocols and Delegation, we can easily have the controller maintain the state.  Here’s I did this.

SideDishTableViewCell -> UITableViewCell

I subclassed the view cell with SideDishTableViewCell (SDTVC).  In SDTVC, I defined a protocol:

// SideDishTableViewCell.h

@protocol SideDishTableViewCellDelegate;
@interface SideDishTableViewCell : UITableViewCell

// define a @property to hold a reference to the delegate
@property (assign, nonatomic) id <SideDishTableViewCellDelegate> delegate;
@end

// define the @protocol here
@protocol SideDishTableViewCellDelegate <NSObject>
@optional
- (int)addItemWithCell:(SideDishTableViewCell *)cell;
- (int)removeItemWithCell:(SideDishTableViewCell *)cell;
@end

Now the SDTVC has a reference to a delegate which we’ll call whenever the user adds or removes items.  One note: I chose to pass the entire SDTVC because it contains bits of data that the controller needs. Another option is to just pass the dish’s ID and have the controller do a bit more work to get its metadata.

Calling the delegate from IBAction

Each of the + and – buttons are tied to IBActions, and the IBAction methods are where we call out to the delegate.

// SideDishTableViewCell.m

// The value 'q' is returned from the controller and is used to update the quantity displayed.  The View does no math.
- (IBAction)increaseQuantity:(id)sender {
    int q = [[self delegate] addItemWithCell:self];
    self.quantityLabel.text = [NSString stringWithFormat:@"%d", q];
}
- (IBAction)decreaseQuantity:(id)sender {
    int q = [[self delegate] removeItemWithCell:self];
    self.quantityLabel.text = [NSString stringWithFormat:@"%d", q];
}

Implementing the protocol from the View Controller

// SideDishViewController.h

#import "SideDishTableViewCell.h"
@interface SideDishViewController : UIViewController <SideDishTableViewCellDelegate>

And the implementation.

// SideDishViewController.m
-(int)addItemWithCell:(SideDishTableViewCell *)cell {
     // Update a collection that holds the dishes and their quantities

     // Return this side dish's quantity
}

Summary

There you have it.  We’ve kept the view very simple to the point it doesn’t even have to do any math.  It simply lets the controller handle the increment/decrement and then just waits for the controller to tell it what the new quantity is.

 

Welcome to the Inner Circle

Inner Circle is a simple game of predicting outcomes of events and comparing yourself against your friends.  I’ve started learning iOS dev and I am building Inner Circle to help me hone my iOS skills.  I have recruited 3 additional players and we just wrapped up the first week.  All questions were NFL selections.  Results are in the pictures below.

I decided to build Inner Circle to give me an opportunity to practice across several areas of iOS development.  This includes:

  • Getting and posting JSON to a server
  • Customizing the Table View to make the question selection cells bow to my command 
  • Using a WebView to view content from the web (this is the tables view)
  • Saving data locally between app restarts (username)
  • App icons – just anything besides the default!
  • And to see if we can make this game fun enough for us to want to continue playing

I’ve also coddled together a simple node.js app to manage all the data.  Feels good to work with Coffeescript again.  So clean!  And Heroku and Mongolab again to the quick and dirty rescue.

And big kudos to Test Flight for making it simple to share builds with your users.  All for free.

Next on the product timeline:

  • Adding the notion of Quizzes, or a grouping of questions
  • This will allow you to work on multiple simultaneous sets of questions
  • Display a selection list of the current open Quizzes

Make your picks

Make your picks

The scores after the first week.

The scores after the first week.

Moving Hopscotch to v3.0

“…to market and promote,
and you better hope
(For what?)
that the product is dope.” – A Tribe Called Quest

What’s happening in Hopscotch 3.0

There comes a time in every product’s lifetime that it needs a major reworking.  Hopscotch.fm has been live for over a year now and I’m happy with where it’s gotten to.  After not working on it much since early 2013, I decided it’s time to give it some love.  I brainstormed what I want to change.  Here are the notes:

Brainstorming notes

Brainstorming notes

The main points I want to focus on in HS 3.0 are:

  • Decouple the music listening from the browsing.  Right now reading about a show auto-plays songs from that show.
  • Use a front end MVC framework.  Leaning strongly towards AngularJS.
  • Responsive UI – the site doesn’t look amazing on a mobile browser.  Good timing as Bootstrap 3 purports to be a mobile first framework now.  Nice.

The layout concept

Looking at the sketch in my notes above, here is the layout I am going to work with.  This screenshot was done using Bootstrap 3.

Overall layout

Overall layout

The layout in code

This is the Jade template using Bootstrap 3:

	
div.container.fullheight

	div.row(style="border:2px")
		div.col-md-8
			h3 Header toolbar
		div.col-md-2
			h3 Social
		div.col-md-2
			h3 City

	div#radio.row.show-grid
		div.col-md-8
			h3 Radio player
		div.col-md-4
			h3 (unused space)

	div#lower.row.show-grid.fullheight
		div#artist.col-md-8.fullheight
			h3 Artist Detail
		div#shows.col-md-4.fullheight
			h3 Upcoming shows

This is the CSS in addition to Bootstrap’s:

div {
	box-shadow: 2px 5px 5px #888888;
}

.fullheight {
	height: 100%;
}

html,body { 
	height:100%; 
}

What’s next

I still want to focus on layout and building the framework to start using AngularJS.  As more concrete objectives, these would be:

  • set up AngularJS.  Create Controller etc and have it load today’s list of shows automatically on page load.  Once this is working, it should be smooth(er) sailing to get fancy.
  • mobile responsive css.  Buzzwords.  Want to get the site looking GREAT on mobile browsers.  Eventually I’d like to get it playing as a continuous radio station on mobile, which I believe can be done with SoundCloud.

Using a single global DB connection in Node.js

“And I’m single, yeah, I’m single,
And I’m single, tonight I’m single,
And I ain’t tripping on nothing, I’m sipping on something,
And my homeboy say he got a bad girl for me tonight” – Lil Wayne

Since Javascript, and in turn Node.js, are single threaded, we can get by with just using one database connection throughout an entire application.  Pretty cool!  We no longer have to open/close/maintain connections and connection pools.  Let’s see how this could be done.

Think Globally, Code Locally

In pure Javascript, any variable you declare that is not defined within another scope (e.g. a function, within an object), will be added to the global scope.  However, because Node wraps each file into its own module with CommonJS, each module does not have direct access to the global scope.  But, Node provides a handy workaround:

global.db = ...

global.varName is accessible from any module.  Perfect.  So we can simply set a db connection on global.db and throw a reuse celebration party!  But before that party, let’s see how we would code this.

Global DB in Express with Mongoose
In this example we will create a single connection to Mongo using the Mongoose library.  In app.js, we define the global connection where we can use the local environment to determine the db uri. We will lazily create it so it will get created only when it is first requested.

// app.js
var express = require('express'); 
var app = express(); 
var mongoose = require('mongoose');

// Configuration 
app.configure('development', function(){ 
  app.set('dburi', 'localhost/mydevdb'); }); 

app.configure('production', function(){ 
  app.set('dburi', 'mongodb://somedb.mongolab.com:1234/myproddb'); }); 

global.db = (global.db ? global.db : mongoose.createConnection(app.settings.dburi));

Now let’s say in another file, we want to create a Mongoose schema for an Alien being.  We simply do it as such:

// otherfile.js
var alienSchema = new Schema({
  planet : { type: String }
});
var Alien = db.model('Alien', alienSchema);

What is happening here is that db.model is looking for db within the current module otherfile.js, not finding it, and then going up the chain till it gets to global.db.  Then boom!  We’ve got aliens.

Closing the connection

Since we want the connection open the entire lifespan of the application, we can simply allow it to terminate upon the application’s termination.  No cleanup needed.  (Caveat: if you know of an explicit way of doing this, perhaps to do some additional cleanup or logging, I’d love to hear about it).

Building a binary search tree in Javascript

“A tree’s a tree. How many more do you need to look at?” – Ronald Reagan

I am reading Secrets of the Javascript Ninja by John Resig and wanted to try out some of the more advanced Javascript concepts.  I also wanted to do something more than just a ‘hello world’ so I decided to build a binary search tree (bst).

Beauty and the BST

There are many articles out there on BST’s so I will skip going into that here.  What I am interested in building is a simple node ‘object’ in JS that can hold references to its left and right children.  To do this, I decided to use the JS prototype functionality.

// Name and value can be set at creation time so are passed into the constructor
function Node(name, value) {
     this.name = name;
     this.value = value;
}

Node.prototype.setLeft = function(left) {
     this.left = left;
}

Node.prototype.setRight = function(right) {
     this.right = right;
}

BST Insertion Logic

Next up is creating the logic that adds a new node to the right place in the BST.  We are not going to get into rebalancing so it is very possible that this tree is waaaay overweighted on one side.  We will live with that and maybe get to that in a future exercise.

// tree is the root node of the tree.  node is the new node to add
// If the new node is greater than tree, then we either add it as the right child if tree does not have a child, otherwise, we call insertNode again but this time passing in tree's right child as the tree parameter.  Similar logic is done if node is less than tree.

function insertNode(tree, node) {
    if (tree) {
        if (tree.value < node.value) {
            if (tree.right) {
                insertNode(tree.right, node);
            } else {
                tree.setRight(node);
            }
        } else {
            if (tree.left) {
                insertNode(tree.left, node);
            } else {
                tree.setLeft(node);
            }
        }
    } else {
        tree = node;
    }
    return tree;
}

Testing the BST Here we do some initial setup in setup, where we add several nodes in various ascending order. Then we print out the tree with printTreeAsc to verify we can walk the tree from lowest to highest, starting from root.

function setup() {
    nodeA = new Node('a', 5);
    nodeB = new Node('b', 12);
    nodeC = new Node('c', 10);
    nodeD = new Node('d', 15);
    nodeE = new Node('e', 20);
    nodeF = new Node('f', 25);
    nodeG = new Node('g', 8);
    nodeH = new Node('h', 3);

    var tree = insertNode(tree, nodeA);
    tree = insertNode(tree, nodeB);
    tree = insertNode(tree, nodeC);
    tree = insertNode(tree, nodeD);
    tree = insertNode(tree, nodeE);
    tree = insertNode(tree, nodeF);
    tree = insertNode(tree, nodeG);    
    tree = insertNode(tree, nodeH);    
}

function printTreeAsc(root) {
    var currNode = root;
    if(currNode.left) {
        printTreeAsc(currNode.left);
    }

    console.log(currNode.value);

    if(currNode.right) {
        printTreeAsc(currNode.right);
    }
}

Running setup() and then printTreeAsc(nodeA) yields:

3
5
8
10
12
15
20
25

It works!
Lastly, how tall is my BST?
BSTs are a fun way to work with algorithms and recursion, so I decided to write a method to calculate the height of the tree. Basically this will return the maximum number of steps from nodeA down to the lowest node. Perfect candidate for recursion!

function calcHeight(node) {
    if (node) {
        return 1 + Math.max(calcHeight(node.left), calcHeight(node.right));
    } else {
        return 0;
    }
}

Result: 5. Passing in nodeA, this gives a result of 5.

Summary
So we got to see the JS prototype feature in action when we build the tree, in insertNode. We also built a simple binary search tree and verified it works by iterating over it in ascending order. And last but not least, we wrote a simple recursive method to determine its height.

Using Node cronjobs to replace Heroku worker dynos

“Time keeps on slipping, slippin…into the future.”  – Steve Miller Band

Hopscotch.fm relies on data that is pulled in from several music related services.  I have created multiple tasks each of which can be run from the command line, like:

node run manualrun getShows sf

This will get the shows for San Francisco. On Heroku, I was using the Scheduler to automatically run this periodically. All good so far.

The Problem

The solution worked great up until I started needing several of these workers, to get shows, to get artist metadata, to get artist songs, cleanup, and more.  Easy enough I thought.  I just added more tasks to the Heroku Scheduler. Except there is a limit to the free tier on Heroku…

The Surprise (the Heroku bill!)

My Heroku bill was over $70!  How did this happen??  Turns out I had exceeded the free monthly hours with all the worker dynos I had been spinning up.  So I needed a solution quick.  I host hopscotch.fm on nodejitsu so I figured why not just use that.

The Solution (cron!)

Enter node-cron. If you’ve ever used Linux/UNIX cron jobs, it’s nearly identical.  The syntax for dates is the same.  All you need to do is specify the function to run.  Here is a cron job in file cronjobs.js that crunches radio stations for a city:

var cronJob = require('cron').CronJob;
var cruncher = require('./cruncher); // internal class that does the crunching
var jobMetroRadio = new cronJob('00 05 00 * * 1-7', (function() {
  console.log('Starting crunching shows.');
  return cruncher.crunchShows(function() {
    return console.log('Finished crunching shows.');
  });
}), (function() {}), true, time.tzset("America/Los_Angeles"));

Then in app.js I just:

require('./cronjobs');

And lastly add these two packages to package.json and install them

cron  // add to package.json
time  // add to package.json
npm install

The Result
I’ve moved all of the tasks over from Heroku Scheduler onto the nodejitsu deployment and everything is running smoothly. Hooray for cron!

How I’m learning CSS

I’ve been hacking at CSS for a while now, copy and pasting and making minor tweaks here and there to get something to work. But I realized there’s only so far hacking can take me and I began feeling like I needed to learn the fundamentals.

So here I’m going to quickly share what I’ve found useful to start learning CSS.

CodeSchool.com

I took the CSS Cross Country course at CodeSchool.com and found it to be a great way to get exposure to all of the main concepts of CSS.  But I still had trouble understanding positioning and floating and so that’s when I turned to:

MDN

Mozilla Developer Network.  Prefix all of your CSS Google searches with mdn.  The pages are elegantly laid out, easy to read and full of information and examples.  This is a reference that I’m sure I’ll be using for a while.  But to really understand positioning and floating, I can’t recommend highly enough:

CSS-Tricks Podcast

I found myself many times ending up on a css-tricks.com article about something CSS related.  Easy to understand articles that go in depth in explaining concepts.  Turns out they’ve got a video podcast on a wide range of topics.  And this is where I finally learned the mystery behind positioning and floating.  I really like learning by watching his split screen where he’s got html, css and the results in three different sections of the page.  It’s pretty informal and Chris seems like a pretty cool dude.

Applying It

That’s not a company or website.  That’s how I will be able to really learn all this stuff.  I’ve got a couple of side projects that I am starting to etch out with CSS.  Had a major breakthrough with floats after a couple of hours of wtf (who knew, fixed heights on uls set the height of the floating div).  Lots of good stuff ahead to learn and I’m pretty excited about it.

And speaking of floats here’s a great explanation of them.

And lastly, some free backgrounds to practice setting your primary div’s background.
mysitemyway.com

We were setting up our own instance of BrowserCMS and it took a few steps to get it up and running. In this post I’m going to recap the steps needed to do this:

Install the basics first
First, get everything up to date first
apt-get update

Next, install rvm to manage the Ruby gems and the active environment
http://beginrescueend.com/

The output will tell you that you need to install a few things. You can copy and run the line that lists the packages required for Ruby.
apt-get install build-essential openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion

Install a few other things including git, patch, Ruby and Rails.

sudo apt-get install git
sudo apt-get install patch
rvm install 1.9.3
apt-get install rails

Install BrowserCMS
Installing the browsercms gems installs all the dependent gems as well
gem install browsercms

I got an error related to building the mysql2 gem. Don’t worry, you just need to install a couple other gems. More info on this here
sudo apt-get install libmysql-ruby
sudo apt-get install libmysqlclient-dev

At this point you’ve installed everything.  All that’s left is to create a new instance.

Create a new BrowserCMS instance

Before you do anything, make sure to set your environment’s RAILS_ENV property.  I set this in the ~/.bashrc but it may be different for you.  Check where you set your env properties in your home folder ls -a should reveal it.  Then be sure to reload the profile or exit and log back into the shell so the env setting takes place.

Here we set it to the development environment.

set RAILS_ENV=development

Next, we want to create the instance and tell it do install it on mysql.  Default would be sqlite.

bcms new apidocs -d mysql

To be sure you've installed everything, let's run bundle install in the newly created app.

cd apidocs

bundle install

Lastly, we want to seed the database with the tables and seed data.  This can be done in one command with rake db:install but I prefer to run each of the three commands that get called by rake db:install, separately.  That way if an error occurs I know exactly which one caused it.  And in this case, there was an error.  First the commands that you run:

rake db:create
rake db:migrate
rake db:seeds

When I ran rake db:seeds with my RAILS_ENV=production, I got an error of:
rake aborted!
private method `rand' called for #

This is because for the production env it is creating a random password and for some reason the rand function on Array is private. Whatever. An easy workaround is I just edited db/browsercms.seeds.rb and added production to the list of checks (line4).  This allows db:seeds to run and then you can update the cmsadmin password via the app later

You can re-run db:seeds to finish the db:install
rake db:seeds

Type in:

rake server

and you should see your server up and running on port 3000.  Put in /cms and you can get to the cms administration portion with your login of cmsadmin / cmsadmin.