Excited to demo Review19 and share my insights on the evolution of next generation web apps. Looking forward to seeing you there!

Details of my talk | Venue | Schedule | JSFoo

Love their excellent Git / GitHub based publishing workflow, excellent way to attract good content from programmers.

Here are the links:
* Node.js and MongoDB – Getting started with MongoJS
* Really simple file uploads with Express

Not just bleeding-edge startups but even medium and large enterprises are leveraging these two technologies to deliver a better experience to their users by build more capable, performant and scalable apps.

So what is Node.js?

Node.js is a platform built on Chrome’s JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.

..and what is MongoDB?

MongoDB is a scalable, high-performance, open source NoSQL database.

This post will cover the basics and get you started with your Node.js + MongoDB app. Make sure you have Node.js installed and MongoDB setup on your developer machine.

Let’s verify your Node.js installation and start the MongoDB server:

$ node -v
$ mongod

Introducing MongoJS
MongoJS is a brilliant little Node.js package that lets you access MongoDB using an API that is extremely similar to MongoDB’s JavaScript shell.

Installing MongoJS
Once Node.js has been setup correctly on your machine, you can use its internal package manager NPM to install MongoJS:

$ npm install mongojs

We can now start building our JavaScript application and connect to our MongoDB server:

// app.js
var databaseUrl = "mydb"; // "username:password@example.com/mydb"
var collections = ["users", "reports"]
var db = require("mongojs").connect(databaseUrl, collections);

The databaseUrl can contain the database server host and port along with the database name to connect to. By default the host is “localhost” and the port is “27017“. If you’re using the default, which is likely on a developer environment, the databaseUrl can contain just the actual database name for our app.

The collections is a set (array) of collections our application uses. It isn’t mandatory but is preferred as it allows us to emulate a MongoDB JavaScript client like API within our Node.js app.

Here’s an example for finding documents within a collection specifically in this case to find all female users.

// app.js
db.users.find({sex: "female"}, function(err, users) {
  if( err || !users) console.log("No female users found");
  else users.forEach( function(femaleUser) {
    console.log(femaleUser);
  } );
});

Notice how our initial query is a near duplication of the corresponding query in MongoDB’s console. In addition to the query, in our Node.js source code (i.e. app.js) we pass a callback function to handle the results of the query.

Node.js implements an event based concurrency paradigm and almost everything is always a callback. This allows your Node.js app to be non-blocking and high performing.

What happens in our specific callback is self-explanatory — we check for errors and results, and if the query returns female users they are logged to the console.

Okay, how do I save a new user in my collection? Exactly how you would do it in the console, your code will look like this:

// app.js
db.users.save({email: "srirangan@gmail.com", password: "iLoveMongo", sex: "male"}, function(err, saved) {
  if( err || !saved ) console.log("User not saved");
  else console.log("User saved");
});

Here’s an example for updating a record / document:

// app.js
db.users.update({email: "srirangan@gmail.com"}, {$set: {password: "iReallyLoveMongo"}}, function(err, updated) {
  if( err || !updated ) console.log("User not updated");
  else console.log("User updated");
});

Okay, now we run this app in the console:

$ node app.js

And there we have it, an incredibly simple quick start for Node.js + MongoDB enthusiasts. Happy coding!

Here’s the question on hand:
When should business, leadership, dev teams and other stakeholders address the scalability question.

It’s a dilema I face and others depend on my advise so I’m pooling in your wisdon.

Allow me to ellaborate the question. We’re building a product and we have two choices:

Choice 1

Go in for a technology that is extremely popular. It has been around in the mainstream for at least half a decade. It’s a good business choice because of the maturity of the environment, the availability of talent / outsourcing firms and positive feedback overall.

However there are some questions over scalability. There are many case-studies where headline grabbing companies have had to abandon this technology and go in for a complete / partial rewrite on another platform.

The argument here goes like this:

“If we use this technology and we face scalability problems few months down the line, it’s a good problem to have because it means we have done really well. So lets go ahead with this for now, even if it means a rewrite six months down the line.”

Choice 2

We can opt for another much younger technology. It has an active community but comparatively less mature environment / toolkits. In comparison to Choice 1, it is harder to find good and affordable talent.

The programming language here is extremely expressive yet heavily misunderstood.

However there are bona fide scaling credentials. It’s been a choice for organisations that are proven technology leaders. And its perhaps a safer bet in the long run on the scalability parameter.

Your call

As a programmer / hacker / geek, Choice 2 is an obvious winner. At least for me its a no-brainer.

However put yourself in a product owner / manager / CTO / decision-making role. Does your opinion change?

Welcoming all your comments, thanks!

Unfortunately all the articles / posts out there are either outdated, too complex or plain wrong. So I did the next most obvious thing – post a question on the mailing list. As always Mr. Holowaychuk was incredibly quick to respond. His answer lead me to do what I should have done in the first place – read the docs.

The upload form

This is the most obvious part of the challenge. You’re probably familiar with this already. Anyway, for the sake of completeness of this article, here it is.

You will need a form in your browser for the file upload. I use Jade to generate my HTML and here how it looks:

form(action="...", method="post", enctype="multipart/form-data")
  input(type="file", name="displayImage")

The form.action will point to a route that handles the file upload. More below.

Accessing the uploaded file

If you’re using recents versions of Node and Express, file uploads are a piece of cake. And I’ll back this claim but before we go any further make sure you’re familiar with routes, requests and responses in Express.

Okay, now let’s justify the “piece of cake” claim. In our file upload route, the req parameter has req.files available. Here’s an example of what the req.files would contain:

{
  displayImage: {
    size: 11885,
    path: '/tmp/1574bb60b4f7e0211fd9ab48f932f3ab',
    name: 'avatar.png',
    type: 'image/png',
    lastModifiedDate: Sun, 05 Feb 2012 05:31:09 GMT,
    _writeStream: {
      path: '/tmp/1574bb60b4f7e0211fd9ab48f932f3ab',
      fd: 14,
      writable: false,
      flags: 'w',
      encoding: 'binary',
      mode: 438,
      bytesWritten: 11885,
      busy: false,
      _queue: [],
      drainable: true
    },
    length: [Getter],
    filename: [Getter],
    mime: [Getter]
  }
}

In the req.files object above, the property displayImage is the name of the file field in your HTML form and req.files will contain one property each for every valid HTML file form field.

The file object contains the type, size and name properties for your server side validations.

Saving the uploaded file

Assuming the file is valid, you use the path property for the next step. The path would typically contain a location in the tmp folder. Your application logic could either require you to access the contents of the file or simply move the uploaded file to another location.

fs.readFile(req.files.displayImage.path, function (err, data) {
  // ...
  var newPath = __dirname + "/uploads/uploadedFileName";
  fs.writeFile(newPath, data, function (err) {
    res.redirect("back");
  });
});

In the fs.readFile callback, we have the data parameter through which we can access the contents of the file. The example above is taken from an application that needed to modify the file and save it in a new location. Thus fs.writeFile is used to write data to the newPath.

If your app needs to simply move the uploaded file without modifying the contents fs.rename can be used as more simpler option.

—

That’s all there is to it. I’ve done file uploads in many server side languages including Python, Java, Scala and PHP and I don’t think its ever been this simple.

So much for JavaScript being labeled as an inferior server side language.

Quick preview:

Objective
How to build the next generation of real-time browser based apps with JavaScript and related technologies.

Description
Ajax is history. After having built Review19 — Review19.com provides next generation real-time collaborative tools — I’d like to share my approach, the technologies used, the architecture followed by a quick demo and code walk through.

Requirements
* Good sense of humor
* Appreciation for random katrina kaif pics inserted between slides
* Knowledge of web app development in general
* More JavaScript (Node, NPM etc.) stack knowledge, the better

If you’re interested, don’t forget to vote.

First things first — what is Review19? Review19 is the next generation, real-time project collaboration tool for web, creative and software teams. Think of it as a minimalist, real-time Pivotal Tracker alternative. It’s like Trello, but caters to a focused target audience.

Give it a shot, try out Review19. If you’re stuck behind a proxy that doesn’t like web sockets, here’s the failover link.

The Technology Stack

Obviously an app can’t be built with a data store alone.

In addition to a MongoDB based storage engine, the APIs and backend for Review19 are built on the Node.js platform and make use of popular frameworks including Express and Socket.io.

The frontend client app — which is under heavy refactoring as we speak — leverages advances in browser capability with respect to JavaScript execution and CSS renderings. The frameworks here include jQuery, jQuery UI and Sugar among others.

Why am I mentioning all of this? The overall technology stack needs to be known to highlight a huge advantage:

JSON. Everywhere!

I’ve been a professional programmer for ten years now and I’ve had the fortune (misfortune?) of working with a diverse set of technologies.

I happen to have dealt with:

• the typical enterprise Java stack – Spring, Hibernate etc.
• recent advances in Java technologies including Scala, LiftWeb, Play! etc.
• built apps in Python using Django, Tornado and Google App Engine
• built apps in PHP and its set of frameworks
• built apps in Adobe Flex working with different types backend APIs

In each of these technologies, a developer has to deal with the mundane task of transforming his model between the client, server and data store. Better development teams would automate this process with the use of ORMs or helper frameworks. However this would still need the inclusion of these dependencies and the configuration of these frameworks.

While developing Review19 with the chosen technologies, things worked out of the box. It was JSON on the server, JSON on the client and, thanks to MongoDB, JSON on the data store. The productivity benefit was huge, especially for an independent developer like me, or small teams with tough deadlines.

The advantage of this cannot be underplayed. It strikes you right away when you build your first proof-of-concept and it stays with you throughout your development cycle.

Flexibility is not just a buzz-word

About 10-12 months ago, I was working on a project for an SF Bay startup that wanted to built an analytics / BI suite running on top of customer databases.

It wasn’t a very complex project, but just a large and poorly managed one. I remember being asked repeatedly about how the database was being versioned. The answer “it was being versioned in the code” wasn’t good enough for the ‘expert’ and his evil overlords.

Hmm.. generating and storing multiple versions of the data model as an .sql file in your source code repository. Sounds familiar? Of course it does. And it sounds just as painful as it sounds familiar. Even if this process is somehow painstakingly automated and devs don’t have to deal with it any more, you still often end up with multiple version of SQL files never being used.

Whatever your application be, the entities / models are bound to evolve with time. Things will change, sometimes so much that they won’t share even the slightest resemblance with the original designs.

With MongoDB, however, the data store adapts seamlessly to your app. There isn’t the slightest need to run scripts, create tables, alter columns etc. on the data store prior to each deployment. This is how it’s done for Review19 and I’d assume for a majority of projects using MongoDB.

This will bring a small morale boost and a large productivity boost for your dev and devops teams.

Node.js + MongoDB = MongoJS

MongoDB is one of the most mature NoSQL options available today and that maturity is proven by the rich set of APIs MongoDB provides that allow you to perform basic and advanced operations on your data.

Furthermore the MongoDB Querying APIs are ridiculously simple, yet powerful and easy to master. In fact its so easy to learn that you wouldn’t miss the very same SQL that you spent years to master.

A majority of MongoDB evangelists and proponents I’ve interacted with swear by this capability. After spending sometime on the MongDB Interactive Shell, I could see what they meant.

I needed to access the same capabilities in my Node.js backend app. Node.js has got a well maintained MongoDB driver and a lot of interesting abractions on top of that driver.

My choice zeroed in on MongoJS and the clinching argument was the MongoJS mission statement:

“A node.js module for mongodb, that emulates the mongodb API as much as possible.”

Your actual executable JavaScript code looks exactly the same to what you do the MongoDB console. How can this not be a #win? I like minimalism and MongoJS was as minimalist as possible.

There are a lot of other options as well for Node.js – MongoDB interaction and it would be best to cover them in detail in a future post.

Speed

Did I mention Review19 is incredibly fast? Almost in all the feedback received, the users mention among other things how fast it is and how it improved their experience.

I understand that application architecture has a role to play in this but one can’t take all the credit for the speed and performance witnessed by Review19 users. One must thank the underlying platforms it was built on which includes Node.js and MongoDB.

I’m sure you’ll find all the MongoDB speed benchmarks you want. They’re just a Google search away. What I intend to highlight here is that in a real world collaborative app which has a single instance running on standard, non-fancy hardware, Node.js and MongoDB are serving the needs brilliantly.

Remember this is a real-time collaboration app where instead of frequent smaller requests, a frontend client has a long lived connection with the backend server. This is exactly the kind of app that would have faced scaling issues few years ago and would have needed a lot of infrastructure both in terms of code and hardware.

Node.js and MongoDB make this economical for independent devs like me without the resources big corps can provide.

Conclusion

I’m extremely pleased with the technology choices made in which of course MongoDB is a critical component. I’ve completed the entire product cycle from conceptualization and design to testing, development and deployments.

Described above are ways MongoDB made this enitre process simpler, kept deployments easy and provided all the features and more needed by Review the app.

Now I look forward the the next set of challenges which include maintaining, scaling and enhancing the system. And I promise to let you know how that goes. Stay tuned to this blog! ;-)

Note – Before reading any further, you should know I am the creator of Review19, a tool quite similar to Trello.

I loved Spolsky’s post, except the title. Trello didn’t seem different at all. Going by a lot of Spolsky’s parameters, Review19 and Trello are very similar.

Here’s trying to shed more light on my claim:

Hosted Only

Review19 – Yes
Trello – Yes

Both Review19 and Trello are hosted only solutions, at least as of now. You cannot buy a license and install privately on your servers.

Continuous Delivery

Review19 – Yes
Trello – Yes

Just like Google Chrome — which still maintains versions — but is released / delivered continuously for all intents and purposes both Review19 and Trello share this development / delivery model.

Inexhaustive Testing

Review19 – Yes
Trello – Yes

Joel’s team at Fog Creek, by choice, don’t exhaustively test Trello before each release. Review19 is being development and maintained by a single individual, i.e. me, it’s not really possible to exhaustively test Review19 either. Sorry about that!

Work in public

Review19 – Yes
Trello – Yes

Trello works in public by offering a public status board. Review19 puts itself out their through a public mailing list and roadmap. Volunteers are added to a private status board.

Get big fast

Review19 – Yes
Trello – Yes

Trello aims to get 100 million users. I’d be an idiot if I didn’t want that for Review19 either. ;-)

Free

Review19 – Yes
Trello – Yes

Plugin architecture

Review19 – Yes
Trello – Yes

The merits of such an architecture are obvious. Review19 uses client and server side JavaScript modules for achieving such an architecture. I’m not sure how Trello does it but we have no reason to doubt Joel’s remark.

Node.js

Review19 – Yes
Trello – Yes

MongoDB

Review19 – Yes
Trello – Yes

Web Sockets

Review19 – Yes
Trello – Yes

How Review19 is different

Horizontal

Review19 – No
Trello – Yes

Trello is for everybody. Review19 is for distributed or colocated web, creative and software teams. Spolsky notes the benefits of being vertical, I agree.

Spolsky wants Trello to be horizontal, I can understand. I did consider making Review19 horizontal but chose against it.

Video conferencing

Review19 – Yes
Trello – No

Review19′s target audience — web, creative and software teams — are often distributed around the world. Offering an in-browser, video conferencing option is a critical feature for Review19.

CoffeeScript

Review19 – No
Trello – Yes

No thanks, I see far too many benefits of staying with JavaScript at least as of now.

APIs top priority?

Review19 – I wish!
Trello – Yes

I wish! I’m only a single guy working full-time on multiple projects to earn my living. Review19 will have APIs, at some point of time!

2011 was also when I had to dig into Scala. Thus I was compelled to implement more solutions in a functional manner.

Like many programmers unfamiliar with Functional Programming or others familiar with Functional Programming but more comfortable with Object Oriented Programming, it was daunting at first.

But persist through and one starts to see the benefits of Functional Programming (or at least understand it enough and execute your projects).

If you’re looking for a quick start in Functional Programming, JavaScript is the perfect language for you. Here’s why:

• Almost all programmers have tweaked and / or written JavaScript code at some point of time — hence there should be a certain familiarity
• JavaScript comes as close to a standardized programming language we’ll get — it’s the only programming language available across all web browsers
• JavaScript comes with a very familiar C like syntax and should be readable to most programmers
• Functions have always been first class members in JavaScript, support for Functional Programming is very good and in many ways JavaScript has been ahead of its time
• JavaScript doesn’t have Java like Class based Object Oriented Programming support so in many ways you’re forced to be Functional in JavaScript

These should be enough reasons, so without meandering any further in the introduction, I present Functional Programming with JavaScript:

Functions in JavaScript

Almost everyone reading would have seen function declarations in JavaScript. However JavaScript lets you explicitly express functions as objects.

In fact, all function declarations are automatically converted to function expressions and hoisted.

By hoisting, the converted function expression is automatically placed in the beginning and thus is available in lines before it has been declared.

JavaScript functions are also different in the sense that each function, regardless of how it is defined, has dynamic arguments and a dynamic context.

A JavaScript function can, of course, access its arguments by their variable names.

By dynamic arguments, I mean, in the function body, we are provided an arguments array which contain all the arguments sequentially in the order they were passed. This means any function can be called with any set of arguments, thus giving us a lot of flexibility.

By dynamic context, I am refering to the this object. Typically the this object defaults sensibly to the context of the function. However it is possible to explicitly call a function with a user defined this / context object.

Notice how functionName.apply() allows us to explicitly set the context and pass arguments.

I must point out that JavaScript presents us with functional scoping instead of block scoping (C, C++ etc.). This usually becomes more relevant when you need to implement Object Oriented patterns in JavaScript but more on that in a later blog post.

Another trick for you is the capability to define self-executing anonymous function. These can be creatively used to implement modular namespaces / packages. Here are some examples:

Higher Order Functions

We know functions in JavaScript are nothing but objects. Does this mean functions can be passed around as arguments to other functions? And can one function return another function?

The answer to both questions is “Yes”. And such functions which taking in other functions are arguments and / or return functions are typically called higher order functions.

In the example below, we will define a higher order function forEach. This function takes in two arguments: a list and an action. The first argument list is an array containing containing objects and the second argument action is a function that will be invoked for each item in list.

The simple example above shows how a named function logItem and an anonymous function are being passed as arguments to the higher order function forEach.

Map / Reduce Implementations

In the example above, we saw how the forEach algorithm was expressed as a function and reused multiple times as an when required. We avoided repeating loops and thus made the code more readable.

This is one of the prime reasons to write functional code. If done correctly, you’ll be able to abstract out complex algorithms, replace it with a nice function call and keep code condensed and readable.

To further stress this idea, we need a better example. For each was too simple an algorithm to abstract. Let’s try implementing higher order functions for Map and Reduce algorithms.

We start with the Map function. What is the map function? Our map function should allow the user to perform a particular action over each item of a list, and then return a result list containing results of actions performed on each list item.

Simple enough, show me the code!

Notice we reuse the forEach function from the previous example. Our map function takes in two arguments, the first one being the mappingFunction that is to be applied on every item of the second argument list.

The implementation for map is straightforward. We prepare an array result, iterate through the list, call the mappingFunction for each item in list, push the result in the result array return it when all done.

As an example, we are calling map function and passing the function doubleIt as the mapperFunction on each item of an anonymous list.

Time for the reduce function. What is the reduce algorithm? The reduce function will be given a combine function, a base value and a list. We need to apply the combine function on the base value and each value in the list and finally return the result. In short, Reduce function takes in a list, combines it and gives us a single result.

This solution again makes use of the previous create forEach function. Our reduce function is simple, it iterates through the list, applies combine on base and each list item, stores the result in base and returns it after completion.

As a usage example, we use reduce to get a count of the negative numbers in a list.

These two examples are fairly simple and still show us how, with functional programming, we can effectively abstract out algorithms as functions, use them as and when required and still retain simplicility and readability in the code.

Closures in JavaScript

If you’ve heard of Functional Programming, there’s a very good chance you’ve also heard about closures. They are often mentioned in the same breath.

But what are closures? Think of closures as a composite of two things:

1. A function being returned by another function
2. A closed environment in which this returned function is forced to execute

This is what a closure is.

In our first example of a closure implementation, we have getAreaFunction which reads the shape argument and returns an appropriate area function.

Notice how getAreaFunction returns an anonymous function. This returned function contains a switch case on shape which is available in the environment.

Finally, of course, we see how to use the closure as a function factory to get area functions for different types of shapes.

In our second closure example, we see how closures are used to implement private class members in JavaScript.

I’ll not explain Object Oriented Programming implementations in JavaScript as part of this post. They aren’t complex but are different, surely worthy of a dedicated post.

If you’re already familiar with OOP concepts and their implementations in JavaScript, note how the class Shape (which is, yes, expressed as a function) contains a private function area and isn’t exposed.

Instead an instance function getArea is available which does internally make use of the private function area.

This is one of the most common use-cases of closures in JavaScript.

Another classic use-case for closures in JavaScript is to avoid the annoying problem that crops up every time you try to set event handlers within a loop.

Consider the following example:

In the for loop, we first assign an onclick handler to all the red boxes. However, thanks to functional scoping, all three red boxes have onclick handlers with redBoxId value 3.

However for green boxes we assign onclick handlers in a slightly different manner. We have a self executing anonymous function which returns another function which then is assigned as the onclick handler.

Due to functional scopes, each onclick handler for the green boxes have correct values for greenBoxId.

I realize this has been a fairly lengthy post. Will end it now. Hopefully this would have served as a good enough introduction to Functional Programming concepts and their corresponding implementations in JavaScript.

All code available on GitHub.

function sayMyName() {
  console.log("!Xobile");
}

Looks very familiar. And is probably the most obvious way programmers have created functions since the beginning of time JavaScript.

In JavaScript, everything is an object. What happens internally is this declaration gets converted into an expression.

What’s an expression? An expression is any statement that produces a value.

Thus, a function expression becomes:

var sayMyName = function() {
  console.log("!Xobile");
};

Function declarations try to hide the fact that the function is indeed an object.

While this may be okay in some cases, it could create confusion when the function is actually being used as a object i.e. being returned, passed around, modified etc.

JavaScript programming (and the programming world in general) is becoming more functional by the day, and it may be a better practice to avoid function declarations altogether in favor of function expressions.

But why? Hoisting.

Function declarations are automatically “hoisted”. The compiler automatically expresses all function declarations at the top of the code regardless of where in the code the function is declared.

This may or may not be a problem depending on the scope-complexity of your code. However it is something that happens automatically, whether you want it or not, and it may be advisable to take such control away from the compiler.

Especially since scoping is very different in JavaScript compared to traditional languages. More on JavaScript scoping in a future blog post.

Review19 is a simple and intuitive project management tool. It is just as suited for teams distributed around the world as it is for colleagues working in the same office.

However it doesn’t try to be a universal, multi-purpose app. Review19 has been built, finetuned and polished specifically for web, creative and software teams.

To begin you create a project and invite members to your project team. Then you (and your team) can add tickets which can be features, enhancements, bugs or chores. Tickets have a description with optional additional details and one or more sub-tasks. In addition, tickets have a priority and can be assigned to a team member.

On the status board, you see all tickets arranged in columns for backlog, scheduled, in progress and complete. Tickets can be dragged across columns and filters can be applied based on user or ticket type.

Oh by the way, while you do all this, the changes you make are synced realtime and your team members can see them and vice-versa. The user interface, built exclusively in HTML5+CSS, keeps this app really simple and intuitive for first time users.

Then there is the “Conference” tab, which lets you engage with your team members with video chats and text chat sessions. The video chat runs in-browser with the flash plugin and doesn’t require any external application be downloaded / installed.

Personally for me, the most exciting part of Review19 is the yet unreleased roadmap which ensures continuous improvements of the core product along with some more incredible features currently unseen in similar products.

Give Review19 a spin and let me know how it works out!

Umm.. what is Review19?

Review19 tries to be a simpler alternative to the likes of Atlassian JIRA and Pivotal Tracker. Its design was driven by the principle of “less is more” yet it can boast of better usability and features.

The new Review19 is a move away from Google App Engine towards Node.js and other open source technologies. The reason is to avoid vendor lock in and leverage new technologies where the Google App Engine SDK lags behind.

However the new Review19 is lot more than just a migration of underlying technology. Presenting some of the more significant updates:

• Video Conferencing – Conduct video chat sessions with your project team within your web browser. Goodbye Skype!
• Kanban + Scrum Board – No complex work flows imposed, just works
• Tickets support features, bugs, chores and enhancements
• Sub-tasking support within tickets
• Numerous usability and user interface enhancements, less glare and intuitive interactions

Give it a spin, try out Review19 at http://www.review19.com

Interested in your feedback, feel free to drop in a message. Join the mailing list for more.

This includes Notifier.js — a Javascript library for Gnome / Growl type non-blocking notifications.

It depends on Jquery. At 68 lines of code, Notifier.js can be classified as minimalist.

Notifier.js supports success, info, warning and error notifications.

Notifications show up at the top-right corner and fade away in 5 seconds (by default). Click and it fades away even sooner.

Multiple notifications at the same time stash up in a vertical list.

Notifier.js has been designed for easy inclusion in any existing small or large project.

Demo | Download (.zip archive, 26.8KBs)

The source code and usage examples can be found on GitHub – https://github.com/Srirangan/notifer.js

Feel free to use it, fork it and provide feedback / pull requests.

Installing Nginx, PHP and MySQL wasn’t a problem and there were tonnes of resources on the internet to help me through. Where I did face a challenge was to get Nginx and PHP to play nicely.

Before I explain what went wrong, I will note the differences in the way Apache2 and Nginx handle PHP.

For Apache2, in many cases, PHP is configured as an internal module. Alternatively it can also be used in CGI mode. However, no matter how it’s configured, it is the web server’s responsibility to interface with PHP (which means having to spawn PHP instances as an when required).

In the case of Nginx, things are very different. Casually put, Nginx acts as a proxy server and will redirect requests to a PHP-CGI server instance running on your box. This means, of course, you are now responsible for spawning a PHP-CGI server instance and configuring Nginx to use it.

Not so bad, so I went ahead and did it. Here’s where the real problem comes in.

After some time, I found that the PHP-CGI instance automatically hung up and needed to be restarted. In the browser this showed up as 502 Bad Gateway errors. I Google’d it for a few hours and couldn’t find a definite solution. However few links did indicate that some PHP modules were the cause for the PHP-CGI crash. The best solution out there I could find was to setup a cron to re-up the PHP-CGI instance every N minutes.

But even if this cron interval is 1 minute, it still leaves a low but significant probability that a user sees the error, refreshes immediately, sees the error again and quits your website. Not acceptable.

My quick fix solution? Reduce this probability. How?

Setup multiple instances of PHP-CGI

$ php-cgi -b 127.0.0.1:9000 &
$ php-cgi -b 127.0.0.1:9001 &
$ php-cgi -b 127.0.0.1:9002 &
$ php-cgi -b 127.0.0.1:9003 &
$ php-cgi -b 127.0.0.1:9004 &

Setup the cron to re-up the PHP-CGI instances if they’re down

$ crontab
*/2  *  *  *  *  php-cgi -b 127.0.0.1:9000
*/2  *  *  *  *  php-cgi -b 127.0.0.1:9001
*/2  *  *  *  *  php-cgi -b 127.0.0.1:9002
*/2  *  *  *  *  php-cgi -b 127.0.0.1:9003
*/2  *  *  *  *  php-cgi -b 127.0.0.1:9004

Configure Nginx to choose from this pool

http {
    ...
    upstream phpfcgi {
        server 127.0.0.1:9000;
        server 127.0.0.1:9001;
        server 127.0.0.1:9002;
        server 127.0.0.1:9003;
        server 127.0.0.1:9004;
    }
    ...
    server {
        server_name        {server name};
        root               {path to document root};

        location / {
            index          index.php;
        }

        location ~\.php$ {
            fastcgi_pass    phpfcgi;
            fastcgi_index   index.php;
            include         fastcgi_params;
            fastcgi_param   SCRIPT_FILENAME $document_root/$fastcgi_script_name;
        }
    }

}

This reduces the probability of the user to repeatedly see the 502 Bad Gateway errors. This can be improved by having the cron trigger a script which checks if the php-cgi process has hung up before attempting to restart it.

But a real solution would be to figure out which PHP module *really* is causing the php-cgi to hang in the first place and fix this bug.

I’ve also stumbled across something called PHP-FPM but I don’t know enough about it to as yet. You might consider checking that out.

Hope this helps.

It took a while to figure out but this is what works for SBT 0.10 or higher:

  sbt update-classifiers

For my Scala projects, I return to using an IDE — IntelliJ IDEA Community (Ultimate is a bit bloated and slow for my liking). The SBT-Idea project links dependencies, sources and docs to your IDEA project with this command:

  sbt gen-idea

Make sure you’ve included SBT Idea as a plugin: https://github.com/mpeltonen/sbt-idea

Here’s a little Node.js utility that does the same. While working on Flex with gedit (or any other text editor), just keep “hotflex” running in the background. It watches all MXML and AS3 files and rebuilds the project on save.

Screenshots

Dependencies

Node.js and NPM are required, refer to – Setup Node.js and NPM on Ubuntu

For notifications on Ubuntu, install libnotify-bin:

    sudo apt-get install libnotify-bin

The Flex SDK is, of course, required and the Flex SDK’s bin folder should be on the $PATH.

Install

From NPM:

    sudo npm install hotflex -g

From GitHub:

    git clone git://github.com/Srirangan/hotflex.git
    cd hotflex
    sudo npm install -g

Configure

You will need to create a hotflex.json file containing the following parameters:

    { "source": "./src/main/flex/main.mxml"
    , "sourceFolder": "./src"
    , "target": "./bin/main.swf"
    , "lib": "./lib"
    }

The lib parameter is optional while the rest are mandatory.

Run

Execute hotflex from the folder containing the hotflex.json file, preferably your project root.

hotflex

Meta

GitHub repository – https://github.com/Srirangan/hotflex
NPM – http://search.npmjs.org/#/hotflex

Todo

• Integration with Adobe Flex Compiler Shell fcsh for faster builds
• Integration for Growl for OSX notifications. Currently notifications work for Ubuntu.

Node.js sources can be downloaded from http://nodejs.org/#download
Alternatively, if you’re a GitHub user, you can clone the repository with:

git clone git://github.com/joyent/node.git

Next step, navigate to the Node.js source root folder and execute the following:

./configure
make
sudo make install

NPM is a package manager for Node similar to Apache Maven, Gradle, SBT etc.

npm is a package manager for node. You can use it to install and publish your node programs. It manages dependencies and does other cool stuff.

Install it with this one line command:

curl http://npmjs.org/install.sh | sh

Here’s attempt to capture a few of these “paradigm shifts” you may have to make. ;-)

Non-blocking + Asynchronous

Line #4 need not execute before line #6. Your design should not require it to do so. Express your implementation as actions and responses i.e. events and handlers.

For instance, in Java you may write:

List users = userService.listUsersByStatus(pending);
users.doSomething();
// Something else happens

In Node, this would be expressed as:

userService.listUsersByStatus(pending, function(users) {
  users.doSomething()
});
// Something else happens

The difference in these two examples is that in Java “something else happens” always executes after users.doSomething().

This is not the case in Node, “something else happens” would typically execute before users.doSomething().

users.doSomething() is called in an anonymous method that serves as a callback for userService.listUsersByStatus().

Less code! Less architecture! Less layers!

JEE tries to over-do everything. Node keeps it very simple. There isn’t any need to ritually create a complex package structure, superstitiously over-design with interfaces, spit out cross generational implementations and follow other bloated JEE patterns.

This doesn’t mean that Node forces you to cramp all your code into a single file either. You can lay it out and distribute it among modules as required. These modules are objects by default and the underlying design principle is minimalism.

For example, a hot topic for debate on Hacker News – ORMs. In Node, you can do an ORM if you want. But is it required?

Given that the persisted objects are to be consumed by a browser, most of the times passing a MongoDB object to browser (JS) object is sufficient. My opinion on ORMs is that, if possible, they should be avoided. However I’m digressing from topic and this post is not about ORMs.

No Application Containers

You don’t deploy a Node webapp on Tomcat. You are much closer to the OS. Typically the entry point of a Node app would listen to a socket and *be* a server.

For example, here’s an ExpressJS webapp:

var app = express.createServer();

app.get('/', function(req, res){
res.send('Hello World');
});

app.listen(3000);

This and its asynchronous nature gives Node apps its high performance, low latencies and real-time capabilities.

IDEs Aren’t Required

I can’t think of working with Java without my trusted JetBrains IntelliJ IDEA. I’ve used Eclipse and NetBeans in the past and they have worked great as well. Without these IDEs for a modern Java Enterprise project, I would be handicapped.

Not so for Node. I’ve been productive without an IDE and while I do miss IntelliJ IDEAs amazing Javascript refactoring features, I gained tremendous performance with a text editor (gedit) and with handy plugins available the trade-off is worth it.

Just a small list of what Java programmers should expect if moving to Node, feel free to add on!

Tweaked it and made it notify for Ubuntu and we have Hotnode with Ubuntu Notifications.

Let’s get started:

Install it..

Dependencies:
sudo apt-get install libnotify-bin

Hotnode:
git clone git://github.com/Srirangan/hotnode.git
cd hotnode
sudo npm install -g

Use it..

Use:
hotnode server.js

Instead of:
node server.js

..there you go!

Things don’t look pretty. In short this is what has happened:

• “Free” quotas have been drastically reduced
• Price of various billiable “units” has seen a sharp rise

The reaction of the Google App Engine customer base has been predictable. Here are some quotes from the official Google App Engine mailing list / group:

“We are going from $5,400/month to $26,500/month (Python) – and this is only one of our apps. We are going to work hard to optimize our application because we really like App Engine, but failing that, we may have to move elsewhere.” — Jason Collins

“Sorry google, but that doesn’t work for me. It makes more sense to go back to django and run the apps on a box in my home. I’ve disabled my apps as of today.” — Johnny G

“Is it really 10 times the original cost.!! Now we will be paying from $200/month to $2000/month under new billing. Just out of our reach, does not make any business sense to stay with GAE.” — smwatch

“I love Google App Engine, but right now I am at a loss for what to do. My current billing is $4.98/day. Under the new system it is $34.41/day, and that is with the 50% frontend discount. The effort that would be involved in porting my app to a different architecture is far beyond what I could accomplish in the next few weeks, and to be honest I don’t want to move off of App Engine.” — Michael Robellard

“It’s 300% for me (3x what I pay now). Most of the cost is datastore writes and reads, followed by instance hours.” — Waleed

“For my app, Aug 26, estimated increase of ~150%.” — Will Xu

“For mine (python): 317% I think I am forced to leave GAE to EC2.” — Raymond C

Is Google setting a precedent for unreliable and unstable cloud billing? Are they pricing themselves out of this market?

If you go by the most popular thread in the Google App Engine community – Keep it short: Who is forced to leave GAE? – then the answer is yes!

The complete announcement from the mailing list:

The first public release of Review19 is out.

Review19 is the next generation, real-time project collaboration tool. It serves as an alternative to Pivotal Tracker, Atlassian JIRA / GreenHopper and other agile / project management tools.

Review19 supports stories, sprints, story board and teams. The interface is simple. No complex rules and work-flows are imposed. Run your project as you like.

Review19 is real-time software. AJAX / asynchronous calls are history. All team members experience real-time collaboration and synchronicity with Review19 eliminating errors and confusions.

Review19 can be accessed at http://www.review19.com/

Review19 is built on Google App Engine, Python, HTML5, JQuery and CSS. It works on recent versions of all real browsers and has been extensively tested on Google Chrome.

Review19 was the result of my disillusionment with Pivotal Tracker which promised so much for so long.

A key design principle for Review19 is for it to be real-time. Real-time as in real-time collaboration. All team members see updates as they happen. Real-time as in Google Docs or Google Wave (RIP).

Simplicity. Pivotal Tracker / Atlassian JIRA etc. were simply too complex to use. They did too much. Review19 does very little. Imposes very few rules and helps you implement almost any workflow (Lean, Agile, Scrum, Kanban etc.)

Please try out Review19. Join the community and share your feedback. Thanks!

DotCloud is a cloud platform built on top of Amazon EC2 that is programmer friendly and not targeted specifically at sysadmins. It will not lock you in to any platform and supports a wide range of open source technologies including MySQL, Node.Js, PHP, PostgreSQL, Python, Redis, Ruby, Java, Perl, SMTP, MongoDB, Solr etc.

DotCloud provides a CLI (Command Line Interface) for performing every action possible including creating new applications, deploying new services, pushing code, running remote commands etc. However Python and related tools are required to get DotCloud CLI working on your Ubuntu machine.

Follow the steps below to get started:

# Install Python (Ubuntu should have it by default)
$ sudo apt-get install python

# Download setuptools and install it with the following command
$ sudo sh ~/Downloads/setuptools-0.6c11-py2.7.egg

# Use easy_install to setup the DotCloud CLI
$ sudo easy_install dotcloud

That’s it! You’re now all set to create your first DotCloud app!

I recently conducted an intensive training session titled “django-kick-start” at the Inphina iBat dated 21st May 2011.

Download source code

This has now changed and am working with Flex 4.5 SDK on IntelliJIDEA + Ubuntu and I feel more productive than ever before. Listed below are steps for you to get started with developing Flex 4.5 applications on IntelliJIDEA:

1. Assumption – You have downloaded Flex SDK 4.5 and have a working installation on IntelliJIDEA. Additionally you have downloaded the Flash debug player (projector) for Linux.
2. Add Flex 4.5 SDK to Global SDKs – Launch the Project Structure window through the file menu or Ctrl + Alt + Shift + S. Under Platform Settings select the SDKs option and add the Flex 4.5 SDK. You should know the path where Flex 4.5 SDK has been extracted.
3. Add Flex 4.5 SDK as a Module Dependency – Under Project Structure select Modules, further select your Flex module and set Flex 4.5 SDK as Module SDK and it to the Dependencies if not already present.
4. Fast Flex Compiler – Open Settings via the file menu or Ctrl + Alt + S, select Flex Compiler and choose the Flex compiler shell for fast Flex compilations. Using Built-in compiler shell did not work for me with Flex 4.5 SDK.
5. Edit configuration with Flash debugger – Create a new Flex run configuration and select the Flash Player Debugger as the Launch Application.
6. All set to go!

If, like me, you need to install Scala 2.8.1 or Scala 2.9.1 on Ubuntu, follow the steps below:

1. Download your desired version of Scala
2. Extract the package in /opt/
3. Edit .bashrc with the command gedit .bashrc
4. Export variable SCALA_HOME: export SCALA_HOME=/opt/scala-2.8.1.final
5. Update PATH to include SCALA binaries: export PATH=$SCALA_HOME/bin:$PATH
6. Save .bashrc, log out and log in
7. Verify the Scala installation: scala -version
8. Install / configure your favorite IDE

You’re ready to go, start hacking..

If so, here are some things you may want to try:

Effectively communicate Scala’s benefits

Scala has a lot of advantages … you know that and I know that but do they know that?

Does your IT leader know that Scala plays really nicely with the existing JVM (and CLR) environments? Do your fellow programmers know of the conciseness of Scala code? Is the older, hardened Java developer aware that most times Java Code *is* perfectly compatible and compilable Scala code?

Has your customer been told of the rapid development, scalability and maintainability edge Scala brings to the engagement?

If stakeholders see the benefit they will recognize the opportunity. And it may very well be that we have earned new allies in our crusade for Scala adoption.

Share the case-studies

The Scala programming language has been proven in high-scalability environments. Twitter got over its stability woes with its migration to Scala. LinkedIn picked Scalatra, a Scala web framework, for its newer modules. Foursquare is built entirely on the Scala Lift Web Framework while Guardian.co.uk recently spoke about their adoption of Scala.

These and many more case studies are out there available helping our understanding of the realistic challenges of Scala adoption, how they were overcome and what are the benefits that have been derived from the overall migration to Scala.

Start with something small

Take the initiative and start with something small and show them how it is done.

For example you may want to introduce a sub module in your existing enterprise project for testing with ScalaTests, build your project utilities with Scala, or implement a simple website / web application with a Scala based web frameworks instead.

Amaze the business with the rapid developement that you’ve achieved, amaze the geeks with your sexy Scala code. Go out there, implement it anyway and bedazzle them.

If things go wrong, you can always apologize and make use of Git or whatever other version control you have. ;-)

Destroy the “It’s not enterprise ready!” argument

You have great IDE support (IntelliJIDEA, NetBeans 7, Eclipse with ScalaIDE 2.9 beta). It integrates very nicely with Maven. You have SBT for projects from scratch. You have books and now you have training programmes. It works seamlessly on your JVM / CLR based environments.

Given the current economic situation, it is pragmatic for business to consider the availability of Scala shops and especially Scala Offshoring partners.

Ask loudly *why* do they specifically think it is not enterprise ready? Chances are they don’t really have a good explanation. That’s not to say Scala doesn’t have its shortcomings, however, what Scala also has is a very enthusiastic and effective community.

Let’s start by creating a Mavenized Scala project, check out Getting Started – Maven Archetype for Scala.

Does the Scala project build successfully? Good.

The next step, add the Querulous dependencies to your project pom.xml file:

<dependencies>
    ..
    <dependency>
      <groupId>com.twitter</groupId>
      <artifactId>querulous</artifactId>
      <version>2.0.1</version>
    </dependency>
    ..
</dependencies>
..
  <repositories>
    ..
    <repository>
      <id>twitter.com</id>
      <url>http://maven.twttr.com</url>
    </repository>
    ..
  </repositories>

In our example, we will migrate a source database boysDb to a destination database girlsDb. No real migration business logic is being applied, think of it as a one-to-one migration.

We start off by declaring QueryEvaluators for our source and destination database schemas:

  val sourceDb = QueryEvaluator("localhost/boysdb", "root", "")
  val destinationDb = QueryEvaluator("localhost/girlsdb", "root", "")

QueryEvaluator object allows us to execute SQL queries on a referenced database.

Next step, get boys from the source database:

val boys = sourceDb.select("select id, name from boys") {
      row => (row.getInt(1), row.getString(2))
    }

boys is a collection of Tuple2[Int, String] containing the id and name attributes of a boy.

For each boy, insert a girl:

boys.foreach(boy => insertGirls(boy))

Let’s now define our insertGirls() method

  def insertGirls(boy: Tuple2[Int, String]): Unit = {
    destinationDb.execute("insert into girls (id, name) values (?, ?)", boy._1, boy._2)
  }

..and we’re done.

Scala + Querulous makes it easy to connect to multiple database instances, makes querying extremely straightforward by exposing SQL directly without unnecessary ORM’ing or configuration / property file mess.

Full source code has been published on GitHub – https://github.com/Srirangan/scala-querulous-simple-migrations

For serious database migrations, do consider Scala Migrations.

This blog post will cover the basics of Scala persistence using Squeryl and the target database system is MySQL.

You will need a Scala project to work with. If you don’t have one already, you can create a Scala project with the Maven archetype.

Step 1 – Define dependencies

Our first dependency is Squeryl. As I blog this, the latest Squeryl version is “0.9.4-RC6″. We need to make this available to our code. Additionally, we need to make available the MySQL Connector as this example persists data into a MySQL database. MySQL Connector is a run-time dependency internally consumed by Squeryl.

    <dependency>
      <groupId>org.squeryl</groupId>
      <artifactId>squeryl_2.8.1</artifactId>
      <version>0.9.4-RC6</version>
    </dependency>
    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
      <version>5.1.15</version>
    </dependency>

If you are using Apache Maven, add the code shown above to the “dependencies” element of your project “pom.xml” file. If you don’t use Maven, make sure the artifacts / JARs are available in your project.

Step 2 – Define entities

Squeryl entities are basic Scala objects.. should we call them POSO’s? Here I’ve defined a BaseEntity and extended it to define a User entity.

package net.srirangan.opensource.squerylexample.entities

import java.sql.Timestamp
import org.squeryl.KeyedEntity

class BaseEntity extends KeyedEntity[Long] {

  val id:Long = 0
  var lastModified = new Timestamp(System.currentTimeMillis)

}

All entities will extend and build on the BaseEntity. The BaseEntity introduces two fields “id” and “lastModified”.

package net.srirangan.opensource.squerylexample.entities

class User(var email:String, var password:String) extends BaseEntity {

  // Zero argument constructor required
  // Squeryl Roadmap says 0.9.5 will not need them :-)
  def this() = this("", "")

}

We’ve now defined the User extending the BaseEntity. In addition to the fields “id” and “lastModified”, a User instance will have “email” and “password” attributes. In addition, we have had to define a zero-argument constructor, however, the Squeryl road-map says that this would not be required from version “0.9.5″ onwards.

Step 3 – Define schema

Squeryl requires us to define a Schema object. Our entity classes (i.e. User) needs to be mapped as tables inside the Schema. Table column properties (unique, index, auto_increment etc.) are also defined here.

package net.srirangan.opensource.squerylexample.schema

import org.squeryl._
import org.squeryl.PrimitiveTypeMode._
import net.srirangan.opensource.squerylexample.entities.User

object Schema extends Schema {

  val users = table[User]

  on(users)(user => declare(
      user.id is (autoIncremented),
      user.email is (unique)
    ))

}

In the code above, we map the User entity to the User table and we’ve declared the email column as unique and the id to be auto incremented. Custom table and column names are possible to define through Squeryl annotations provided, check out the Squeryl docs.

Step 4 – Start database session

Needless to mention, we will need an active database session if we are to perform CRUD operations on the database. Here’s how I’ve done it:

  import org.squeryl.Session
  import org.squeryl.SessionFactory
  import org.squeryl.adapters.MySQLAdapter

  val databaseUsername = "squeryl-example"
  val databasePassword = "squeryl-example"
  val databaseConnection = "jdbc:mysql://localhost:3306/squeryl-example"

  def startDatabaseSession():Unit = {
    Class.forName("com.mysql.jdbc.Driver")
      SessionFactory.concreteFactory = Some(() => Session.create(
          java.sql.DriverManager.getConnection(databaseConnection, databaseUsername, databasePassword),
          new MySQLAdapter)
        )
  }

Step 5 – Generate schema

Once we’ve initialized the database session, we can generate the database schema as shown below:

  import org.squeryl.PrimitiveTypeMode._
  import net.srirangan.opensource.squerylexample.schema.Schema

  startDatabaseSession()

  transaction {
    Schema.create
    println("Created the schema")
  }

Step 6 – Insert and Update

  import net.srirangan.opensource.squerylexample.entities.User

  transaction {
    val user1:User = new User("user1@domain.com", "oldPassword")
    Schema.users.insert(user1)
    println("Inserted user1")

    user1.password = "newPassword"
    Schema.users.update(user1)
    println("Updated user1")
  }

Step 7 – Select

    transaction {
      val queriedUser:User = Schema.users.where(user => user.id === 2L).single
      println(queriedUser.id + " -- " + queriedUser.email)
    }

The entire squeryl-example source code is available at github.com/Srirangan/squeryl-example

More examples on Squeryl.org.

I set out to build this crawler.

Prior experience had made aware of distributed computing technologies such as Hadoop and GridGain, so I knew there was my start. I immediately picked GridGain over Hadoop. Pretty obvious reasons: More examples, better support etc.

My next choice was a programming language. Java was the obvious choice but I took a risk and chose Scala. GridGain’s support for Scala and abundance of examples made this choice a bit easier. A quick, unofficial definition for those unaware: Scala is an Objective-Functional programming language that is very attractive to programmers and has proved itself in high-scalability situations (Twitter, LinkedIn, FourSquare etc.)

Note – I am new to Scala and my Scala code may look more Java like than Functional. I’m still learning and future examples should be better. “Awesomeness of Scala code” not a valid parameter to judge this blog post!

Professional etiquette (and NDAs + lawyers) will not allow me to share exact details of this crawler. After all, it is not my intellectual property. But for the sake of this example I will consider my target to be a simple web crawler that would be used by search engines to index the content on the internet.

What would our web crawler do?

1. Start at some base URL
2. Index content of this URL
3. Search for more URLs to index
4. Repeat 2 & 3 for these new URLs

This blog post will not get into the operational logic of loading a URL, extracting keywords, adding to index, extracting URLs etc. That I believe has been done to death. Alternatively I will look at how to scale up the crawling process using Scala and GridGain.

Those already familiar with GridGain, for the sake of this example I would request you to merge the concepts of a GridTask and a GridJob. Here we will create custom GridTasks which have one corresponding, unique custom GridJob.

Our GridTask-GridJob Pairs will be:

• LoadUrlDataTask, LoadUrlDataJob
• IndexKeywordsTask, IndexKeywordsJob

Much of the game is being played in LoadUrlDataJob. Its role is envisioned as follows:

1. Make HTTP request to URL
2. Gather response data from URL
3. Trigger IndexKeywordsTask for URL data
4. Fetch new URLs from URL data
5. Trigger LoadUrlDataTask for new URLs

While the rest have simple roles:

• LoadUrlDataTask = Return one LoadUrlDataJob
• IndexKeywordsTask = Return one IndexKeywordsJob
• IndexKeywordsJob = Parse data and index keywords

In other words, an IndexKeywords job would index keywords and die. In contrast, a LoadUrlData job would trigger exactly one IndexKeyword job and trigger potentially multiple LoadUrlData jobs.

Let’s look at the sources:

Complete Mavenized sources for Scala GridGain SimpleWebCrawler can be found on GitHub.com – https://github.com/Srirangan/simplewebcrawler

A quick look at the role of LoadUrlDataJob and we know that this needs to scale and scale big. Here is a visualization showing three levels of LoadUrlData wherein each LoadUrlDataJob spawns three other LoadUrlDataJobs and one IndexKeywords Job.

GridGain takes care of this seamlessly and divides the tasks among available nodes without any configuration or instruction. Here are screenshots showing three nodes of GridGain, one inside my IDE while other two on the console.

Is this a perfect web crawler? No. Far from it. For one, you need to control its spawn-rate else your machine will die. :-)

But it is an example that does showcase the power of GridGain and the ease with which Scala / Scalar can leverage it.

But first .. What is Scala? Here’s how Scala-lang.org defines it:

“Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages, enabling Java and other programmers to be more productive. Code sizes are typically reduced by a factor of two to three when compared to an equivalent Java application.”

Let’s get started. First, you need Maven pre-installed and configured. If you don’t have it ready, stop right here.

Start the console and enter the following archetype:generate command:

$ mvn archetype:generate -DarchetypeGroupId=org.scala-tools.archetypes -DarchetypeArtifactId=scala-archetype-simple

This enters the interactive mode and asks you to enter the project groupId, artifactId, version and other project coordinates. After which it generates a Scala project.

This should have been it, but unfortunately it is not. You need to edit the generated pom.xml and uncomment the repositories and pluginRepositories elements.

<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>

<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>

Don’t ask me why they are commented in the first place, but as of March 5th 2011 they are and it sucks that they are.

By now we have avoided “dependency not found, build error”  and thankfully this is the only modification required.

You should now be able to successfully build the project..

$ mvn compile

..and get started with real programming.