Srirangan

Programmer + Product Developer

How I built a real-time collaborative app with MongoDB

January 14th, 2012

This is an account of how Review19 was built in just a couple of weeks and how MongoDB was an excellent choice for this genre of browser based, real-time collaborative apps.

First things first — what is Review19? Review19 is the next generation, real-time project collaboration tool for web, creative and software teams. Think of it as a minimalist, real-time Pivotal Tracker alternative. It’s like Trello, but caters to a focused target audience.

Give it a shot, try out Review19. If you’re stuck behind a proxy that doesn’t like web sockets, here’s the failover link.

The Technology Stack

Obviously an app can’t be built with a data store alone.

In addition to a MongoDB based storage engine, the APIs and backend for Review19 are built on the Node.js platform and make use of popular frameworks including Express and Socket.io.

The frontend client app — which is under heavy refactoring as we speak — leverages advances in browser capability with respect to JavaScript execution and CSS renderings. The frameworks here include jQuery, jQuery UI and Sugar among others.

Why am I mentioning all of this? The overall technology stack needs to be known to highlight a huge advantage:

JSON. Everywhere!

I’ve been a professional programmer for ten years now and I’ve had the fortune (misfortune?) of working with a diverse set of technologies.

I happen to have dealt with:

  • the typical enterprise Java stack – Spring, Hibernate etc.
  • recent advances in Java technologies including Scala, LiftWeb, Play! etc.
  • built apps in Python using Django, Tornado and Google App Engine
  • built apps in PHP and its set of frameworks
  • built apps in Adobe Flex working with different types backend APIs

In each of these technologies, a developer has to deal with the mundane task of transforming his model between the client, server and data store. Better development teams would automate this process with the use of ORMs or helper frameworks. However this would still need the inclusion of these dependencies and the configuration of these frameworks.

While developing Review19 with the chosen technologies, things worked out of the box. It was JSON on the server, JSON on the client and, thanks to MongoDB, JSON on the data store. The productivity benefit was huge, especially for an independent developer like me, or small teams with tough deadlines.

The advantage of this cannot be underplayed. It strikes you right away when you build your first proof-of-concept and it stays with you throughout your development cycle.

Flexibility is not just a buzz-word

About 10-12 months ago, I was working on a project for an SF Bay startup that wanted to built an analytics / BI suite running on top of customer databases.

It wasn’t a very complex project, but just a large and poorly managed one. I remember being asked repeatedly about how the database was being versioned. The answer “it was being versioned in the code” wasn’t good enough for the ‘expert’ and his evil overlords.

Hmm.. generating and storing multiple versions of the data model as an .sql file in your source code repository. Sounds familiar? Of course it does. And it sounds just as painful as it sounds familiar. Even if this process is somehow painstakingly automated and devs don’t have to deal with it any more, you still often end up with multiple version of SQL files never being used.

Whatever your application be, the entities / models are bound to evolve with time. Things will change, sometimes so much that they won’t share even the slightest resemblance with the original designs.

With MongoDB, however, the data store adapts seamlessly to your app. There isn’t the slightest need to run scripts, create tables, alter columns etc. on the data store prior to each deployment. This is how it’s done for Review19 and I’d assume for a majority of projects using MongoDB.

This will bring a small morale boost and a large productivity boost for your dev and devops teams.

Node.js + MongoDB = MongoJS

MongoDB is one of the most mature NoSQL options available today and that maturity is proven by the rich set of APIs MongoDB provides that allow you to perform basic and advanced operations on your data.

Furthermore the MongoDB Querying APIs are ridiculously simple, yet powerful and easy to master. In fact its so easy to learn that you wouldn’t miss the very same SQL that you spent years to master.

A majority of MongoDB evangelists and proponents I’ve interacted with swear by this capability. After spending sometime on the MongDB Interactive Shell, I could see what they meant.

I needed to access the same capabilities in my Node.js backend app. Node.js has got a well maintained MongoDB driver and a lot of interesting abractions on top of that driver.

My choice zeroed in on MongoJS and the clinching argument was the MongoJS mission statement:

“A node.js module for mongodb, that emulates the mongodb API as much as possible.”

Your actual executable JavaScript code looks exactly the same to what you do the MongoDB console. How can this not be a #win? I like minimalism and MongoJS was as minimalist as possible.

There are a lot of other options as well for Node.js – MongoDB interaction and it would be best to cover them in detail in a future post.

Speed

Did I mention Review19 is incredibly fast? Almost in all the feedback received, the users mention among other things how fast it is and how it improved their experience.

I understand that application architecture has a role to play in this but one can’t take all the credit for the speed and performance witnessed by Review19 users. One must thank the underlying platforms it was built on which includes Node.js and MongoDB.

I’m sure you’ll find all the MongoDB speed benchmarks you want. They’re just a Google search away. What I intend to highlight here is that in a real world collaborative app which has a single instance running on standard, non-fancy hardware, Node.js and MongoDB are serving the needs brilliantly.

Remember this is a real-time collaboration app where instead of frequent smaller requests, a frontend client has a long lived connection with the backend server. This is exactly the kind of app that would have faced scaling issues few years ago and would have needed a lot of infrastructure both in terms of code and hardware.

Node.js and MongoDB make this economical for independent devs like me without the resources big corps can provide.

Conclusion

I’m extremely pleased with the technology choices made in which of course MongoDB is a critical component. I’ve completed the entire product cycle from conceptualization and design to testing, development and deployments.

Described above are ways MongoDB made this enitre process simpler, kept deployments easy and provided all the features and more needed by Review the app.

Now I look forward the the next set of challenges which include maintaining, scaling and enhancing the system. And I promise to let you know how that goes. Stay tuned to this blog! ;-)

How Trello is *not* different

January 7th, 2012

Joel Spolsky, a figure lot of us look up to, made a very interesting post about Trello and how it is different.

Note – Before reading any further, you should know I am the creator of Review19, a tool quite similar to Trello.

I loved Spolsky’s post, except the title. Trello didn’t seem different at all. Going by a lot of Spolsky’s parameters, Review19 and Trello are very similar.

Here’s trying to shed more light on my claim:

Hosted Only

Review19 – Yes
Trello – Yes

Both Review19 and Trello are hosted only solutions, at least as of now. You cannot buy a license and install privately on your servers.

Continuous Delivery

Review19 – Yes
Trello – Yes

Just like Google Chrome — which still maintains versions — but is released / delivered continuously for all intents and purposes both Review19 and Trello share this development / delivery model.

Inexhaustive Testing

Review19 – Yes
Trello – Yes

Joel’s team at Fog Creek, by choice, don’t exhaustively test Trello before each release. Review19 is being development and maintained by a single individual, i.e. me, it’s not really possible to exhaustively test Review19 either. Sorry about that!

Work in public

Review19 – Yes
Trello – Yes

Trello works in public by offering a public status board. Review19 puts itself out their through a public mailing list and roadmap. Volunteers are added to a private status board.

Get big fast

Review19 – Yes
Trello – Yes

Trello aims to get 100 million users. I’d be an idiot if I didn’t want that for Review19 either. ;-)

Free

Review19 – Yes
Trello – Yes

Plugin architecture

Review19 – Yes
Trello – Yes

The merits of such an architecture are obvious. Review19 uses client and server side JavaScript modules for achieving such an architecture. I’m not sure how Trello does it but we have no reason to doubt Joel’s remark.

Node.js

Review19 – Yes
Trello – Yes

MongoDB

Review19 – Yes
Trello – Yes

Web Sockets

Review19 – Yes
Trello – Yes

How Review19 is different

Horizontal

Review19 – No
Trello – Yes

Trello is for everybody. Review19 is for distributed or colocated web, creative and software teams. Spolsky notes the benefits of being vertical, I agree.

Spolsky wants Trello to be horizontal, I can understand. I did consider making Review19 horizontal but chose against it.

Video conferencing

Review19 – Yes
Trello – No

Review19′s target audience — web, creative and software teams — are often distributed around the world. Offering an in-browser, video conferencing option is a critical feature for Review19.

CoffeeScript

Review19 – No
Trello – Yes

No thanks, I see far too many benefits of staying with JavaScript at least as of now.

APIs top priority?

Review19 – I wish!
Trello – Yes

I wish! I’m only a single guy working full-time on multiple projects to earn my living. Review19 will have APIs, at some point of time!

An Introduction to Functional Programming in JavaScript

December 25th, 2011

For me, in many ways, 2011 has been the year of the rediscovery of JavaScript. I have been programming in JavaScript for many years (like most other programmers) but now I’ve come to appreciate JavaScript the language a little better.

2011 was also when I had to dig into Scala. Thus I was compelled to implement more solutions in a functional manner.

Like many programmers unfamiliar with Functional Programming or others familiar with Functional Programming but more comfortable with Object Oriented Programming, it was daunting at first.

But persist through and one starts to see the benefits of Functional Programming (or at least understand it enough and execute your projects).

If you’re looking for a quick start in Functional Programming, JavaScript is the perfect language for you. Here’s why:

  • Almost all programmers have tweaked and / or written JavaScript code at some point of time — hence there should be a certain familiarity
  • JavaScript comes as close to a standardized programming language we’ll get — it’s the only programming language available across all web browsers
  • JavaScript comes with a very familiar C like syntax and should be readable to most programmers
  • Functions have always been first class members in JavaScript, support for Functional Programming is very good and in many ways JavaScript has been ahead of its time
  • JavaScript doesn’t have Java like Class based Object Oriented Programming support so in many ways you’re forced to be Functional in JavaScript

These should be enough reasons, so without meandering any further in the introduction, I present Functional Programming with JavaScript:


Functions in JavaScript

Almost everyone reading would have seen function declarations in JavaScript. However JavaScript lets you explicitly express functions as objects.

In fact, all function declarations are automatically converted to function expressions and hoisted.

By hoisting, the converted function expression is automatically placed in the beginning and thus is available in lines before it has been declared.


JavaScript functions are also different in the sense that each function, regardless of how it is defined, has dynamic arguments and a dynamic context.

A JavaScript function can, of course, access its arguments by their variable names.

By dynamic arguments, I mean, in the function body, we are provided an arguments array which contain all the arguments sequentially in the order they were passed. This means any function can be called with any set of arguments, thus giving us a lot of flexibility.

By dynamic context, I am refering to the this object. Typically the this object defaults sensibly to the context of the function. However it is possible to explicitly call a function with a user defined this / context object.

Notice how functionName.apply() allows us to explicitly set the context and pass arguments.


I must point out that JavaScript presents us with functional scoping instead of block scoping (C, C++ etc.). This usually becomes more relevant when you need to implement Object Oriented patterns in JavaScript but more on that in a later blog post.

Another trick for you is the capability to define self-executing anonymous function. These can be creatively used to implement modular namespaces / packages. Here are some examples:


Higher Order Functions

We know functions in JavaScript are nothing but objects. Does this mean functions can be passed around as arguments to other functions? And can one function return another function?

The answer to both questions is “Yes”. And such functions which taking in other functions are arguments and / or return functions are typically called higher order functions.

In the example below, we will define a higher order function forEach. This function takes in two arguments: a list and an action. The first argument list is an array containing containing objects and the second argument action is a function that will be invoked for each item in list.

The simple example above shows how a named function logItem and an anonymous function are being passed as arguments to the higher order function forEach.


Map / Reduce Implementations

In the example above, we saw how the forEach algorithm was expressed as a function and reused multiple times as an when required. We avoided repeating loops and thus made the code more readable.

This is one of the prime reasons to write functional code. If done correctly, you’ll be able to abstract out complex algorithms, replace it with a nice function call and keep code condensed and readable.

To further stress this idea, we need a better example. For each was too simple an algorithm to abstract. Let’s try implementing higher order functions for Map and Reduce algorithms.

We start with the Map function. What is the map function? Our map function should allow the user to perform a particular action over each item of a list, and then return a result list containing results of actions performed on each list item.

Simple enough, show me the code!

Notice we reuse the forEach function from the previous example. Our map function takes in two arguments, the first one being the mappingFunction that is to be applied on every item of the second argument list.

The implementation for map is straightforward. We prepare an array result, iterate through the list, call the mappingFunction for each item in list, push the result in the result array return it when all done.

As an example, we are calling map function and passing the function doubleIt as the mapperFunction on each item of an anonymous list.

Time for the reduce function. What is the reduce algorithm? The reduce function will be given a combine function, a base value and a list. We need to apply the combine function on the base value and each value in the list and finally return the result. In short, Reduce function takes in a list, combines it and gives us a single result.

This solution again makes use of the previous create forEach function. Our reduce function is simple, it iterates through the list, applies combine on base and each list item, stores the result in base and returns it after completion.

As a usage example, we use reduce to get a count of the negative numbers in a list.

These two examples are fairly simple and still show us how, with functional programming, we can effectively abstract out algorithms as functions, use them as and when required and still retain simplicility and readability in the code.


Closures in JavaScript

If you’ve heard of Functional Programming, there’s a very good chance you’ve also heard about closures. They are often mentioned in the same breath.

But what are closures? Think of closures as a composite of two things:

  1. A function being returned by another function
  2. A closed environment in which this returned function is forced to execute

This is what a closure is.

In our first example of a closure implementation, we have getAreaFunction which reads the shape argument and returns an appropriate area function.

Notice how getAreaFunction returns an anonymous function. This returned function contains a switch case on shape which is available in the environment.

Finally, of course, we see how to use the closure as a function factory to get area functions for different types of shapes.


In our second closure example, we see how closures are used to implement private class members in JavaScript.

I’ll not explain Object Oriented Programming implementations in JavaScript as part of this post. They aren’t complex but are different, surely worthy of a dedicated post.

If you’re already familiar with OOP concepts and their implementations in JavaScript, note how the class Shape (which is, yes, expressed as a function) contains a private function area and isn’t exposed.

Instead an instance function getArea is available which does internally make use of the private function area.

This is one of the most common use-cases of closures in JavaScript.


Another classic use-case for closures in JavaScript is to avoid the annoying problem that crops up every time you try to set event handlers within a loop.

Consider the following example:

In the for loop, we first assign an onclick handler to all the red boxes. However, thanks to functional scoping, all three red boxes have onclick handlers with redBoxId value 3.

However for green boxes we assign onclick handlers in a slightly different manner. We have a self executing anonymous function which returns another function which then is assigned as the onclick handler.

Due to functional scopes, each onclick handler for the green boxes have correct values for greenBoxId.


I realize this has been a fairly lengthy post. Will end it now. Hopefully this would have served as a good enough introduction to Functional Programming concepts and their corresponding implementations in JavaScript.

All code available on GitHub.

JavaScript – Function Declarations vs. Function Expressions

December 24th, 2011

Here’s an example of a function declaration in JavaScript:

function sayMyName() {
  console.log("!Xobile");
}

Looks very familiar. And is probably the most obvious way programmers have created functions since the beginning of time JavaScript.

In JavaScript, everything is an object. What happens internally is this declaration gets converted into an expression.

What’s an expression? An expression is any statement that produces a value.

Thus, a function expression becomes:

var sayMyName = function() {
  console.log("!Xobile");
};

Function declarations try to hide the fact that the function is indeed an object.

While this may be okay in some cases, it could create confusion when the function is actually being used as a object i.e. being returned, passed around, modified etc.

JavaScript programming (and the programming world in general) is becoming more functional by the day, and it may be a better practice to avoid function declarations altogether in favor of function expressions.

But why? Hoisting.

Function declarations are automatically “hoisted”. The compiler automatically expresses all function declarations at the top of the code regardless of where in the code the function is declared.

This may or may not be a problem depending on the scope-complexity of your code. However it is something that happens automatically, whether you want it or not, and it may be advisable to take such control away from the compiler.

Especially since scoping is very different in JavaScript compared to traditional languages. More on JavaScript scoping in a future blog post.

Why you must use Review19 for your project

December 7th, 2011

Last week I rolled out a new preview release of Review19.

Review19 is a simple and intuitive project management tool. It is just as suited for teams distributed around the world as it is for colleagues working in the same office.

However it doesn’t try to be a universal, multi-purpose app. Review19 has been built, finetuned and polished specifically for web, creative and software teams.

To begin you create a project and invite members to your project team. Then you (and your team) can add tickets which can be features, enhancements, bugs or chores. Tickets have a description with optional additional details and one or more sub-tasks. In addition, tickets have a priority and can be assigned to a team member.

On the status board, you see all tickets arranged in columns for backlog, scheduled, in progress and complete. Tickets can be dragged across columns and filters can be applied based on user or ticket type.

Oh by the way, while you do all this, the changes you make are synced realtime and your team members can see them and vice-versa. The user interface, built exclusively in HTML5+CSS, keeps this app really simple and intuitive for first time users.

Then there is the “Conference” tab, which lets you engage with your team members with video chats and text chat sessions. The video chat runs in-browser with the flash plugin and doesn’t require any external application be downloaded / installed.

Personally for me, the most exciting part of Review19 is the yet unreleased roadmap which ensures continuous improvements of the core product along with some more incredible features currently unseen in similar products.

Give Review19 a spin and let me know how it works out!