Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

July 25 2013

21:07

Two-Factor Auth Using Authy

It feels like every day you read about a new security breach on a website, in many cases involving weak passwords. It’s a known fact that users are notoriously lax when it comes to choosing their passwords, so there needs to be a better way of providing secure logins to your site. That’s where two-factor security comes in. It aims to complement your existing login scheme by also providing an alternative verification method (generally a cellphone) to validate that a user is, who they say they are. It’s a scheme in use by top sites such as Twitter and GMail and has proven to be very reliable in minimizing intrusions via weak passwords.

In this screencast, I’ll introduce you to a great service called Authy which provides a platform to build two-factor authentication right into your site.

Download Video

July 15 2013

22:04

Building Apps With the Yeoman Workflow

What Is Yeoman?

Trick question. It’s not a thing. It’s this guy:

yo

Basically, he wears a top hat, lives in your computer, and waits for you to tell him what kind of application you wish to create. As an example, scaffolding a web application would look something like this:

yo-webapp

The first thing that comes to mind is OMG so dreamy. Second, thanks, bro.

All we did was tell him what we wanted and he replied with specific questions to give us even more control.

Let’s go back a step, though. Not every new computer comes with a Yeoman pre-installed. He lives in the NPM package repository. You only have to ask for him once, then he packs up and moves into your hard drive. Make sure you clean up, he likes new and shiny things.

He’s a person with feelings and opinions, but he’s very easy to work with. If you think he’s too opinionated, he can be easily convinced.

Let’s take a second to break apart what that yo webapp command, from the previous screenshot,really did.

yo

This is an OS X, Linux, and Windows friendly system-wide command that scours your hard drive for any installed “generators,” then gives them control based on the next argument:

webapp

This is actually a separate plug-in, or “generator,” called generator-webapp. Yeoman recognizes other generator-____ Node modules, which opens the door for using Backbone, AngularJS, and countless other you-name-it generators.

Something important to take away from this is, it’s the generator-webapp module that prompts us with questions. The same goes for any other generators we install. They are maintained by the community, not necessarily the Yeoman team members themselves.

By using Yeoman, you’re not saying “I want to do things your way, master. bow bow,” without having any control. It’s actually quite the opposite. What you’re really saying is, “I want to make an application that follows best practices that have been discovered by frequent users and contributors of the web development community.”

Seriously, you have to say it just like that, or it won’t work.

Should you prefer to do something differently than what he gives you, you simply change the code that was generated for you, or even go to the source of the “generator” itself, and send in your contribution.


Friendship

Our buddy, yo has some buddies of his own, and thinks you’ll all get along over endless tea and smiles. If you haven’t heard of Grunt or Bower, here’s a quick summary of what these give us:

Grunt

Grunt is a JavaScript-based task runner, that does the dirty stuff. Like yo, it also provides a base set of functionality, then allows the community to share their own plug-ins, or “tasks” that help accomplish common things. When you scaffold your application with yo webapp, Grunt and some hand-picked tasks will come along, which accomplish things like running your website in a local development environment, concatenating and minifying your code, optimizing your images, and much more. Tasks are run through the command line, by typing grunt server, grunt build, grunt test, and many more.

Tasks are defined and customized in a Gruntfile.js file, which lives in the root directory of your project. Check it out to see what Yeoman set up for you.

Bower

Nobody likes going to GitHub or random developers’ sites to download a .zip of a JavaScript tool. Like when fetching a Node package with npm install ___, Bower lets you say bower install ___. The component is then saved in a directory of your choosing, generally, app/bower_components/ for Yeoman-generated apps. Assuming you wanted jQuery, you would run the bower install query command, then include the relevant file inside of your HTML file. In our case, <script src="bower_components/jquery/jquery.js"></script>


A Typical Application

Let’s get wild. It’s time to create an app.

Real quick though, find your nearest terminal and make sure you have yo installed globally:

$ npm install -g yo

Create a folder we can play around in, then run:

$ yo webapp

Here’s what should have happened:

  • A whole buncha stuff.

Did it? Good!

To prevent you from scrolling up through all of the text that was just spit out at you, here’s an overview:

yo-webapp-long

The new web application was scaffolded and your Bower components and NPM packages were automatically installed.

Open all this new stuff in your favorite editor, and we’ll look over what we have.

├─ app/
│  ├─ images/
│  │  ├─ glyphicons-halflings.png
│  │  └─ glyphicons-halflings-white.png
│  ├─ scripts/
│  │  ├─ vendor/
│  │  │  └─ bootstrap.js
│  │  ├─ app.js
│  │  ├─ hello.coffee
│  │  └─ main.js
│  ├─ styles/
│  │  └─ main.css
│  ├─ .htaccess
│  ├─ 404.html
│  ├─ favicon.ico
│  ├─ index.html
│  └─ robots.txt
│
├─ node_modules/
│  ├─ so/
│  ├─ many/
│  └─ packages/
│
├─ test/
│  ├─ spec/
│  │  └─ test.js
│  ├─ .bowerrc
│  ├─ bower.json
│  └─ index.html
│
├─ .bowerrc
├─ .editorconfig
├─ .gitattributes
├─ .gitignore
├─ .jshintrc
├─ bower.json
├─ Gruntfile.js
└─ package.json

If you take anything away from this article, let it be the beautiful file/folder text representation above. That just took a whole Mountain Dew of my time.

Back on track. What you’re looking at is the most common application structure a Yeoman generator will produce.

  • app/ is where your pure, non-compiled, non-minified source code lives.
  • app/scripts/ is where your JavaScript goes. You’re free to create sub-directories and even use CoffeeScript if that’s your cup of tea. That didn’t make sense. Again. You’re free to use TeaScript if that’s your cup of coffee. Nope.
  • app/styles/ is where your CSS goes. Again, sub-directories, LESS, Sass, whatevs.
  • app/index.html is the non-minified version of index.html that will eventually be squashed and delivered to the client. More on that later.
  • Gruntfile.js has all of the build, server, and test tasks defined.

At this point, yo has done his job. He’s given you everything you need to launch a production-ready web application. Let’s now shift our focus to what Grunt tasks he’s pre-configured for us.

grunt build

Running grunt build takes your app/ source code files and turns them into a distributable application, which ends up in dist/.

That dist/ folder is what you feed to your server. dist/ will have it’s own index.html, with references to minified and concatenated dist/scripts and dist/styles, and optimized dist/images. Your users will appreciate this. Your phone-card, dial-up users will really appreciate this.

Behind the scenes, grunt build is a task that runs several sub-tasks. One of those is grunt-usemin, which looks for blocks inside of your app/index.html, like this:

app/index.html

<!-- build:js scripts/main.js -->
<script src="bower_components/jquery/jquery.js"></script>
<script src="scripts/main.js"></script>
<!-- endbuild -->

After your grunt build task completes, you will end up with this:

dist/index.html

<script src="scripts/c155266f.main.js"></script>

It sucked those scripts up, concatenated, minified, and even prefixed them with unique hashes to prevent browsers from caching outdated versions. Quite powerful.

That’s one of the shining features about using Yeoman. Instead of manually defining what you want your build process to do each time you create an application, you can just place some trust in Yo and your chosen generator. Together, they’ll wire you up with everything you need to launch a production-ready application.

grunt server

Now that you’ve seen what type of work grunt build will do when your application is complete, you should probably start working on your application! We’ll create one together in just a sec, but first let’s see what kind of workflow we’ll have. Like grunt build, grunt server uses several other Grunt tasks to make development as easy as it can be.

Try it out:

yo-webapp-grunt-server

The aforementioned “several other Grunt tasks” are:

  • clean: Yeoman stores some stuff in a .tmp folder. That will be wiped out.
  • coffee: Compiles your CoffeeScript files from app/scripts.
  • compass: Compiles your Sass files from app/styles.
  • connect: Creates a local server, watches for changes to your source files, then triggers a reload in your browser.
  • open: Opens the server instance, typically localhost:9000 in your browser.

Make an edit or two in the source files to see the change reflected in the browser. Like I said above, this is about as easy as it can be. It just works.


Let’s App It Up!

I of course meant appetizers. Grab some cheese sticks, then meet me in a little bit.

Wash your hands!


Let’s Create an Application

To get a feel for some other Yeoman generators, let’s try out Backbone. We’ll create a simple To Do app, use Bower for our dependencies, and introduce you to a real-life workflow with Yeoman.

$ Sound good? (Y/n)

I’ll assume you entered “Y”. We ride! But first:

# install the Backbone generator:
$ npm install -g generator-backbone

# make another play directory, then do these things:
$ yo backbone

     _-----_
    |       |
    |--(o)--|   .--------------------------.
   `---------´  |    Welcome to Yeoman,    |
    ( _´U`_ )   |   ladies and gentlemen!  |
    /___A___   '__________________________'
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

Out of the box I include HTML5 Boilerplate, jQuery, Backbone.js and Modernizr.

Would you like to include Twitter Bootstrap for Sass? (y/N) No
Would you like to include RequireJS (for AMD support)? (y/N) No

Open the new app in your editor. Things should feel quite familiar after our experience with the web app generator. You still have an app directory, with scripts/, styles/ and an index.html.

Before we start editing files, run:

$ grunt server

As we talked about earlier, this starts the server, sets up watches on our files, blah blah yipsie-doodle. Your browser should open, and you should be greeted with:

‘Allo, ‘Allo!

Well, shoot, we have to keep that. It’s just so nice. However, let’s clear out the other stuff.

index.html

<div class="container">
    <div class="hero-unit">
        <h1>'Allo, 'Allo!</h1>
        <section id="todo-app">
            <!-- Where our To Do app will go -->
        </section>
    </div>
</div>

When you save, your browser will refresh, and there we have it! Just a simple, warm “‘Allo, ‘Allo”.

Let’s get ourselves a game plan. We know we’re going to create a To Do app, but what might that look like? Will we need any other libraries to help us?

Hmm.

It’s been at least 4 seconds, and I haven’t heard any answers.

Alright, I’m gonna grab another Dew after that file tree drank my last one. I’ll let you know if I think of anything.


To Do: Set Up Our File Structure

B3. A terrible slot in a vending machine for a carbonated drink. Fizz, foam, disaster.

While I was in the bathroom washing my hands, I had a vision.

[ Add a New To Do ] ← input

checkbox
- clicking will draw a line through the title of the to do item
 ↓
[x] To Do Item #1
[ ] To Do Item #2
    ↑ title
      - double clicking will trigger an "edit" mode

Or…

vision-big

Let’s set ourselves up with a structure that will bring this vision to life.

generator-backbone came with some secret weapons: sub-generators. yo backbone scaffolded our application, but flip back to your terminal and check out what these guys can do:

todo-generate-models

Check out your index.html:

<!-- build:js scripts/main.js -->
<script src="scripts/main.js"></script>
<script src="scripts/templates.js"></script>
<script src="scripts/collections/todos-collection.js"></script>
<script src="scripts/models/todo-model.js"></script>
<script src="scripts/views/todos-view.js"></script>
<script src="scripts/views/todo-view.js"></script>
<!-- endbuild -->

How ’bout that! It not only created and placed files in relevant directories, it even included them in your HTML for you.

I’ve created a repository for our To Do application&mbdash; go check it out. We’ll take a glance at the files together, but please refer to the repository to get the full code.

scripts/main.js

/*global backboneApp, $*/

window.backboneApp = {
    Models: {},
    Collections: {},
    Views: {},
    Routers: {},
    init: function () {
        new this.Views.TodosView({
            collection: new this.Collections.TodosCollection()
        });
    }
};

$(document).ready(function () {
    backboneApp.init();
});

Thoughts

The Backbone generator is establishing some good practices you can use right out of the box. It took the name of your directory, in my case “backboneApp”, and exposed an object literal to hold the Models, Collections, and other Backbone objects we may create.

The generator also incorporates JSHint into your app’s build process, making sure your code is of the highest, most consistent quality. You are encouraged to customize your preferences inside the .jshintrc file in the root of your project’s directory.

Finally, $(document).ready will call backboneApp.init, which creates a TodosCollection, then passes it into a TodosView. I’ll go over these in more detail soon.

scripts/collections/todos-collection.js

/*global backboneApp, Backbone*/

backboneApp.Collections.TodosCollection = Backbone.Collection.extend({

    localStorage: new Backbone.LocalStorage('backbone-generator-todos'),

    initialize: function () {
        this.model = backboneApp.Models.TodoModel;
    }

});

Thoughts

If we want our To Do app to be somewhat usable, we have to store our To Do items somewhere. There’s a handy Backbone adapter you may be familiar with called Backbone.LocalStorage. It will intercept Backbone’s calls to the default remote backend and use your browser’s window.localStorage instead.

We know we’ll need the Backbone.LocalStorage adapter, but where should we go to get it? Idea! Idea!

We haven’t made much use of Bower directly. When our application was scaffolded, Bower was used behind the scenes to grab Modernizr, Twitter Bootstrap, jQuery, Underscore, and Backbone. But, what if we want to add in another JavaScript dependency?

Go back to your favorite terminal and try this:

$ bower search backbone
bower-search-backbone

Ok, wow. That’s… a lot. Maybe we should narrow that down.

$ bower search backbone.localstorage
Search results:

    backbone.localStorage git://github.com/jeromegn/Backbone.localStorage.git

There we go. Now we just have to install it.

$ bower install backbone.localStorage --save
bower cloning git://github.com/jeromegn/Backbone.localStorage.git
bower cached git://github.com/jeromegn/Backbone.localStorage.git
bower fetching backbone.localStorage
bower checking out backbone.localStorage#v1.1.4
bower installing backbone.localStorage#v1.1.4

When working with multiple developers, it can be troublesome assuring everyone has the correct dependencies and matching versions. By using --save above, we are telling Bower to remember this new dependency, then write about it in our bower.json file. When another developer clones your project, they just have to run bower install to download every dependency, keeping everyone in sync. That’s why app/bower_components is listed in your .gitignore file. Gone are the days of bloated repositories!

Now that Bower has awesomed all over our application, go into app/index.html and update the scripts/vendor.js comment block:

<!-- build:js scripts/vendor.js -->
<script src="bower_components/jquery/jquery.js"></script>
<script src="bower_components/underscore/underscore.js"></script>
<script src="bower_components/backbone/backbone.js"></script>
<script src="bower_components/backbone.localStorage/backbone.localStorage.js"></script>
<!-- endbuild -->

When you save the file, your browser will refresh and you’ll have the new library ready to use. More specifically, TodosCollection will have it ready to use.

scripts/collections/todo-model.js

/*global backboneApp, Backbone*/

backboneApp.Models.TodoModel = Backbone.Model.extend({

    defaults: {
        title: '',
        completed: false
    },

    toggle: function () {
        this.save({
            completed: !this.get('completed')
        });
    }

});

Thoughts

This is a pretty basic Backbone Model. We set some default properties for our To Do items and define a toggle function, simply used to switch between a “Complete” or “Incomplete” state.

scripts/views/todos-view.js

/*global backboneApp, Backbone, JST*/

backboneApp.Views.TodosView = Backbone.View.extend({

    el: '#todo-app',

    template: JST['app/scripts/templates/todos.ejs'],

    events: { /* ... */ },

    initialize: function () { /* ... */ },

    render: function () { /* ... */ },

    createTodo: function () { /* ... */ },

    addTodoItem: function () { /* ... */ },

    addAllTodoItems: function () { /* ... */ }

});

Thoughts

This is our most robust Backbone View, so to see the definitions to these various properties and methods, please refer to the repository.

However, here are a couple key things:

el: '#todo-app'

This selector matches that <section id="todo-app"></section> element we created in our index.html file. This will be our primary View.

template: JST['app/scripts/templates/todos.ejs']

This little JST thing snuck in when we said yo backbone:view ____. When our View’s JavaScript file was created, the Backbone sub-generator created a matching template file for us: app/scripts/templates/todos.ejs.

These .ejs template files will define our Views’ HTML. When we run our app with grunt server or grunt build, our template files will be crushed together into a JavaScript object, JST. When our view file says template: JST['path/to/view/template.ejs'], this is referring to that object.

scripts/templates/todos.ejs

<form class="input-append">
    <input type="text" id="new-todo" placeholder="What do you need to do today?">
    <input type="submit" class="btn" value="Submit">
</form>
<ul>
    <!-- Where our To Do items will go -->
</ul>

Thoughts

Because we answered “Yes” to including Twitter Bootstrap for Sass when we scaffolded our application, I’ve added a couple of class names to pretty up our app. Feel free to style to your heart’s content in the styles/main.scss file.

styles/main.scss

@import 'sass-bootstrap/lib/bootstrap';

.hero-unit {
    margin: 50px auto 0 auto;
    width: 300px;
}

form {
    margin-top: 10px;
}

ul,
li form {
    margin: 0;
    padding: 0;
}

ul {
    list-style: none;
}

li form {
    display: none;
}

.editing {
    span {
        display: none;
    }

    form {
        display: inline-block;
    }
}

input:checked ~ span {
    text-decoration: line-through;
}

Thoughts

Sass is pretty cool.

Also, it’s pretty cool that the browser still reloads when you make a change to your Sass files. If you’ve used Sass before, you know it can be a hassle to get a productive development environment set up quickly. Out of the Yeoman box, you’re editing, watching, and reloading with none of the aforementioned hassle. Smiley face.

scripts/views/todo-view.js

/*global backboneApp, Backbone, JST*/

backboneApp.Views.TodoView = Backbone.View.extend({

    tagName: 'li',

    template: JST['app/scripts/templates/todo.ejs'],

    events: {
        'click input[type="checkbox"]': 'toggle',
        'dblclick span': 'toggleEdit',
        'submit form': 'toggleEdit'
    },

    initialize: function () { /* ... */ },

    render: function () { /* ... */ },

    toggle: function () { /* ... */ },

    toggleEdit: function () { /* ... */ }

});

Thoughts

This TodoView will represent an individual item. It will be an <li> with some custom functionality handling click, double click, and submit events, enabling a user to edit and save a To Do item.

scripts/templates/todo.ejs

<input type="checkbox" <% if (completed) { %>checked<% } %>>
<form>
    <input type="text" value="<%= title %>">
</form>
<span>
    <%= title %>
</span>

Thoughts

Simple enough. We’re using some basic Underscore templating to spit out values and toggle a checked state on our checkbox.


To Do: Do It Again

Our To Do application is actually done! It’s quite basic in functionality, but you should have a sense of how natural it is to develop an application using Yeoman and his Generator buddies. And even though the functionality is basic, none of the techniques we used to get here were “basic.” We’re using smart, efficient libraries (Sass, Backbone, Underscore) with a finely-tuned development process (Grunt, LiveReload, Compass), and it took us only a few terminal commands.

If you’re like me, you probably want to stop with the To Do stuff and start making your own applications. If you want to go play around, go for it! When you’re done generating like a crazy person, come back and let’s ship our To Do app.


To Do: Ship It

Let’s put this thing in the water and see if she floats! Do NOT put your computer in the water. Wait, would a MacBook Air float? No, probably not. Hmm…

That was a dangerous paragraph. Let’s just get our app ready for production, safe and dry.

grunt server has been amazing, but it’s time to meet his brother, grunt build. We talked about him a bit earlier, but let’s go over a few more details.

Here is what the grunt build task is defined as in your Gruntfile.js:

grunt.registerTask('build', [
    'clean:dist',    // Clears out your .tmp/ and dist/ folders
    'coffee',        // Compiles your CoffeeScript files (if any)
    'createDefaultTemplate', // Creates a JS file that sets up your JST object
    'jst',           // Compiles your `scripts/templates/` files
    'compass:dist',  // Compiles your Sassiness
    'useminPrepare', // Looks for those <!-- special blocks --> in your HTML
    'imagemin',      // Optimizes your images!
    'htmlmin',       // Minifies your HTML files
    'concat',        // Task used to concatenate your JS and CSS
    'cssmin',        // Minifies your CSS files
    'uglify',        // Task used to minify your JS
    'copy',          // Copies files from .tmp/ and app/ into dist/
    'rev',           // Creates unique hashes and re-names your new JS/CSS files
    'usemin'         // Updates the references in your HTML with the new files
]);

So, that thing is pretty legit. All of these tasks are defined inside of Gruntfile.js, so feel free to poke and tweak around to customize your application’s build. It’s highly likely you won’t need to do any customization at all, but it’s there if you need to.

Oh, one other thing. grunt build is actually wrapped inside of another task.

grunt

Simply running grunt will execute the default task:

grunt.registerTask('default', [
    'jshint',
    'test',
    'build'
]);

Those first two tasks, jshint and test are easy to overlook when rushing an app out the door, but are very important.

JSHint

The jshint task will consult with your .jshintrc file to learn your preferences, then scan through all of your JS files to make sure your rules are abided by. To get the full run down of your options with JSHint, check the JSHint documentation.

Test

The test task looks like this:

grunt.registerTask('test', [
    'clean:server',
    'coffee',
    'createDefaultTemplate',
    'jst',
    'compass',
    'connect:test',
    'mocha'
]);

It basically does enough to create and serve your application for your test framework, Mocha, to execute your tests.

Oh crap, tests.

Next door to your app/ and dist/ directories, this little test/ buckaroo has been waiting for our attention. Aww.

If you open that up, you’ll see test/ has its own bower.json and index.html, as well as a spec/ directory. Your tests will have some dependencies of their own, the Chai Assertion Library and Mocha testing framework.

Expand that spec/ directory and you’ll see a test.js file that looks something like this:

/*global describe, it */
'use strict';

(function () {
    describe('Give it some context', function () {
        describe('maybe a bit more context here', function () {
            it('should run here few assertions', function () {

            });
        });
    });
})();

Ok, looks like we could use a pull request to correct some grammar. Anybody?

If you haven’t written your own tests before, you’ll see terms like describe, it, before, beforeEach, after, and afterEach pop up. describe is a wrapper for a group of related tests, ____Each are optional functions that will execute before or after your test(s), and each it is a specific test.

Try running a grunt test to see all the magic unfold.

todo-grunt-test

You should play around and see if you can write some tests for our To Do application. A few ideas for test cases might be:

  • Does creating a new To Do item get saved in localStorage?
  • Does a new To Do item’s title get trimmed (removing extra whitespace)?
  • When editing a To Do item, does deleting the title, then saving remove the To Do item from localStorage?

There’s only one more thing to do.

Press Enter

$ grunt

You should see our favorite words: Done, without errors.


Finding Yeoman

Yeoman is still quite young; he just turned one! Things are pretty great now and they’re only going to get better. However, like all one year olds, Yeoman is still learning to walk without falling, and talk without drooling. You just might run into a bug or two. In times like these, think of him like your cute little nephew. He needs positive role models in his life, so help him learn!

That got real children’s book-y, real fast. I’ll grow it up a little: there are bugs and we need your help to squash the doody out of them (I said “a little”). Even if it’s not a bug, but you’re like, “I know a MUCH faster Grunt plug-in this generator could use,” report it to the appropriate generator’s issue tracker.

If you want to learn some more about Yeoman or just get to know the team, you’ll find us peppered all over the following sites.

If you’re just plain stuck, try one of the following resources for a helping hand.

Yeoman is just one piece of the entire stack— NPM, Node, Grunt, and Bower. It can be intimidating if you’re unfamiliar with these, but it is crucial not to fear the curve! Learning will need to happen, and like always, it will probably need to happen the hard way before it really sticks.

Psst, if you’re using sudo before every command, run, don’t walk, to Node and NPM in 30 Seconds. There, you’ll find several scripts you can run to give control back to your user account. It will also help you install Node and NPM if you’re starting from scratch.


Yo’ Next Application – Will You Yo?

Like all tools, I believe Yeoman is something every developer should try. If you give it a shot and find it’s not suitable for your task, I and the rest of the team would love to hear why. If you need a buddy to help you with your project, come find me. I’m always available around the links above, or just ping me on Twitter. I’m @stephenplusplus or Stephen Sawchuk.

me

Nice to meet you.

Sponsored post
feedback2020-admin
14:56

July 05 2013

20:28

Developing Google Chrome Extensions

It’s no secret that my favorite browser is Google Chrome. I like it because it’s fast, reliable, it doesn’t crash (very often), and it looks good. There’s also something else which I find even more valuable. It’s the fact that you can build an extension for it using only HTML, CSS, and JavaScript. I always support such products, products that are open to the community and Chrome happens to be one of these products. If you need something and it is not yet implemented, you are free to develop it yourself.

So at the end of this article you will find a working Chrome extension which uses most of the techniques explained below. You can download the final example using the download source code button at the top of this page.


Why You Should Write Your Own Extension

I always encourage people to use better tools to speed up their workflow. The software which we use should help us, we should not have to fight with it. Developing extensions/plugins for your favorite editor or browser helps not only you, but also other programmers, who sooner or later will be in the same situation. If something is missing, you can build it yourself and with Chrome this is really easy. Modeling your environment around your needs is key to being highly productive.


Developing & Testing Out Your Extensions

Thankfully there is a way to test your extension without having to upload it to Chrome’s web store. In your browser’s address bar, just type in:

chrome://extensions

Make sure that you check Developer mode and click the Load unpacked extension… button. Then simply select the folder from your hard disk which contains the extension’s files.

extensionspanel

Architecture

Here’s a diagram of the architecture for a Chrome extension:

architecture

And now let’s take a closer look at each element within the architecture.

Manifest

The entry point of your extension is the manifest.json file. It should contain a valid JSON object. For example:

{
    "name": "BrowserActionExtension",
    "version": "0.0.1",
    "manifest_version": 2,
    "browser_action": {
        "default_title": "That's the tool tip",
        "default_popup": "popup.html"
    }
}

The required properties are name, version, and manifest_version. The version can be anywhere from one to four, dot-separated integers. It’s something which is used by Google’s autoupdate system. That’s how it knows when to update your extension. The value of the manifest_version should be the integer 2.

The manifest could contain other properties depending on what kind of extension you need, but I’ll describe only those which I find to be more interesting.

Background Pages

Every extension has an invisible background page which is run by the browser. There are two types – persistent background pages and event pages. The first one is active, all of the time. The second is active only when it is needed. Google encourages developers to use event pages, because this saves memory and improves the overall performance of the browser. However, it’s good to know that this is also where you should put your main logic and initialization. Normally the background page/script plays the role of a bridge between the other parts of the extension.

Here is how you should describe it in the manifest:

"background": {
    "scripts": ["background.js"],
    "persistent": false/true
}

As you may have guessed, if the persistent property is false then you are using event pages. Otherwise, you are working with a persistent background page.

Content Script

If you need access to the current page’s DOM, then you have to use a content script. The code is run within the context of the current web page, which means that it will be executed with every refresh. To add such a script, use the following syntax.

"content_scripts": [
    {
        "matches": ["http://*/*", "https://*/*"],
        "js": ["content.js"]
    }
]

Keep in mind that the value of matches determines for which pages your script will be used. Read more about matches patterns here.

User Interface

There are several ways to build the UI of your extension. Here are the four most popular.

Browser Action

Most developers use the browser_action property to build their plugins. Once you set it, an icon representing your extension will be placed on the right side of the address bar. Users can then click the icon and open a pop-up which is actually HTML content controlled by you.

browseraction

The manifest files should contain the following data:

"browser_action": {
    "default_icon": {
        "19": "icons/19x19.png",
        "38": "icons/38x38.png"
    },
    "default_title": "That's the tool tip",
    "default_popup": "popup.html"
}

The default_title is a little tool tip which is shown when the user mouses over your icon. default_popup is actually the HTML file which is loaded inside the pop-up. There is also a badge which you can place over your icon. You can do that inside of your background script. For example:

chrome.browserAction.setBadgeText({text: "yeah"});

This was the code which I used to produce the image above.

Page Action

The page_action property is similar to the browser action, but the icon is shown inside the address bar:

pageaction

The interesting thing here is that your icon is hidden initially, so you should decide when to show it. For example, in the image above, the RSS icon will be shown only if the current page contains a link to the RSS feed. If you need to see your icon all the time, it is good to use browser_action directly.

To add the page action, type the following code inside your manifest:

"page_action": {
    "default_icon": {
        "19": "images/icon19.png",
        "38": "images/icon38.png"
    },
    "default_title": "Google Mail",
    "default_popup": "popup.html"
}

Unlike the browser action’s icon, the page action’s icon doesn’t have badges.

DeveloperTools

I use DeveloperTools a lot and it’s nice that Chrome offers a method for adding new tabs to these tools. The first thing you should do is add an HTML page which will be loaded when the panel is opened:

"devtools_page": "devtools.html"

There’s no need to put any HTML inside the page, except for linking in a JavaScript file, which will create the tab:

<script src="devtools.js"></script>;

And then include the following code inside the devtools.js file:

chrome.devtools.panels.create(
    "TheNameOfYourExtension", 
    "img/icon16.png", 
    "index.html",
    function() {

    }
);

Now the above code will add a new tab with a name of TheNameOfYourExtension and once you click on it the browser will load index.html inside the DeveloperTools.

Omnibox

The omnibox is the keyword which is shown inside Chrome’s address bar. For example, if you add the following property into your manifest:

"omnibox": { "keyword" : "yeah" }

And then add the code below, inside your background script:

chrome.omnibox.onInputChanged.addListener(function(text, suggest) {
    suggest([
      {content: text + " one", description: "the first one"},
      {content: text + " number two", description: "the second entry"}
    ]);
});
chrome.omnibox.onInputEntered.addListener(function(text) {
    alert('You just typed "' + text + '"');
});

You should be able to type yeah inside the address bar. Then you should see something like this:

omnibox

Pressing tab will produce the following screen:

omnibox2

Of course using the chrome.omnibox API, you could catch the user’s input and react to that input.

APIs

There are bunch of different things which you can do in your extension. For example, you can get access to the user’s bookmarks or history. You can move, create tabs or even resize the main window. I strongly recommend to check out the documentation to get a better idea of how to accomplish these tasks.

What you should know is that not all APIs are available in every part of your extension. For example, your content script can’t access chrome.devtools.panels or the script in your DeveloperTools tab can’t read the page’s DOM. So, if you’re wondering why something is not working, this could be why.

Messaging

As I mentioned above, you don’t always have access to the API that you want to use. If that’s the case, then you should use message passing. There are two types of messaging – one-time requests and long-lived connections.

One-Time Requests

This type of communication happens only once. I.e. you send a message and wait for an answer. For example, you could place the following code in your background script:

chrome.extension.onMessage.addListener(function(request, sender, sendResponse) {
    switch(request.type) {
        case "dom-loaded":
            alert(request.data.myProperty);
        break;
    }
    return true;
});

Then use the code from below in your content script:

window.addEventListener("load", function() {
    chrome.extension.sendMessage({
        type: "dom-loaded", 
        data: {
            myProperty: "value"
        }
    });
}, true);

And this is how you can get information about the current page’s DOM and use it inside your background script, which normally doesn’t have access to this data.

Long-Lived Connections

Use this type of messaging if you need a persistent communication channel. Inside your content script place the following code:

var port = chrome.runtime.connect({name: "my-channel"});
port.postMessage({myProperty: "value"});
port.onMessage.addListener(function(msg) {
    // do some stuff here
});

And then in the background script, use this:

chrome.runtime.onConnect.addListener(function(port) {
    if(port.name == "my-channel"){
        port.onMessage.addListener(function(msg) {
            // do some stuff here
        });
    }
});

Override Pages

Overriding pages is a nice way to customize your browser. You’re also able to substitute some of the default pages in Chrome. For example you can create your own history page. To do that, add in the following code snippet:

"chrome_url_overrides" : {
    "<page to override>;": "custom.html"
}

The possible values of <page to override> are bookmarks, history, and newtab. It’s kinda cool to have a fresh new tab page.


An Example Extension

To wrap up this article I decided to include a simple example, so you can get a better understanding of the whole picture. This example extension uses most of the things that I described above to simply set a #F00 background color for all of the divs in the current page. Feel free to download the source code using the button at the top of this article.

The Manifest File

Of course I started with the manifest file:

{
    "name": "BrowserExtension",
    "version": "0.0.1",
    "manifest_version": 2,
    "description" : "Description ...",
    "icons": { "16": "icons/16x16.png", "48": "icons/48x48.png", "128": "icons/128x128.png" },
    "omnibox": { "keyword" : "yeah" },
    "browser_action": {
        "default_icon": { "19": "icons/19x19.png", "38": "icons/38x38.png" },
        "default_title": "That's the tool tip",
        "default_popup": "browseraction/popup.html"
    },
    "background": {
        "scripts": ["background.js"],
        "persistent": false
    },
    "chrome_url_overrides" : {
        "newtab": "newtab/newtab.html"
    },
    "content_scripts": [{
        "matches": ["http://*/*", "https://*/*"],
        "js": ["content.js"]
    }],
    "devtools_page": "devtools/devtools.html"
}

Keep in mind that you can organize your files into folders. Also, pay attention to the version property. You should update this property every time you want to upload your extension into the web store.

Background Script

// omnibox
chrome.omnibox.onInputChanged.addListener(function(text, suggest) {
    suggest([
      {content: "color-divs", description: "Make everything red"}
    ]);
});
chrome.omnibox.onInputEntered.addListener(function(text) {
    if(text == "color-divs") colorDivs();
});

// listening for an event / one-time requests
// coming from the popup
chrome.extension.onMessage.addListener(function(request, sender, sendResponse) {
    switch(request.type) {
        case "color-divs":
            colorDivs();
        break;
    }
    return true;
});

// listening for an event / long-lived connections
// coming from devtools
chrome.extension.onConnect.addListener(function (port) {
    port.onMessage.addListener(function (message) {
        switch(port.name) {
            case "color-divs-port":
                colorDivs();
            break;
        }
    });
});

// send a message to the content script
var colorDivs = function() {
    chrome.tabs.getSelected(null, function(tab){
        chrome.tabs.sendMessage(tab.id, {type: "colors-div", color: "#F00"});
        // setting a badge
        chrome.browserAction.setBadgeText({text: "red!"});
    });
}

The first few lines get the user’s action from the omnibox. After that, I set a one-time request listener, which will accept the message from the browser action icon.

The next snippet is a long-lived connection with the devtools tab (it’s not absolutely necessary to use a long-lived connection for this, I did it just for educational purposes). Using these listeners, I’m able to get the input from the user and send it to the content script, which has access to the DOM elements. The key point here was to first select the tab which I wanted to manipulate and then send a message to it. Lastly, I put a badge on the extensions icon.

Browser Action

We start with our popup.html file:

// popup.html
<script type="text/javascript" src="popup.js"></script>
<div style="width:200px">
    <button id="button">Color all the divs</button>
</div>

Then we create the popup.js file:

// popup.js
window.onload = function() {
    document.getElementById("button").onclick = function() {
        chrome.extension.sendMessage({
            type: "color-divs"
        });
    }
}

The pop-up contains a single button and once the user clicks it, it sends a message to the background script.

DeveloperTools

window.onload = function() {
    var port = chrome.extension.connect({ name: "color-divs-port" });
    document.getElementById("button").onclick = function() {
        port.postMessage({ type: "color-divs"});
    }
}

For the DeveloperTools, we’re doing almost the same thing here as we did in the pop-up, the only difference is that I used a long-lived connection.

Content Script

chrome.extension.onMessage.addListener(function(message, sender, sendResponse) {
    switch(message.type) {
        case "colors-div":
            var divs = document.querySelectorAll("div");
            if(divs.length === 0) {
                alert("There are no any divs in the page.");
            } else {
                for(var i=0; i&lt;divs.length; i++) {
                    divs[i].style.backgroundColor = message.color;
                }
            }
        break;
    }
});

The content script listens for a message, selects all the divs on the current page, and changes their background color. Pay attention to the object which I attached the listener to. In the content script that’s chrome.extension.onMessage.

Customizing the New Tab Page

The last thing that this extension does is customize the new tab page. We can do that easily just by pointing the newtab property to the newtab/newtab.html file:

"chrome_url_overrides" : {
    "newtab": "newtab/newtab.html"
}

Keep in mind that you can’t create a replica of the default new tab page. The idea of this feature is to add in a completely different functionality. Here is what Google says:

Don’t try to emulate the default New Tab page. The APIs necessary to create a slightly modified version of the default New Tab page — with top pages, recently closed pages, tips, a theme background image, and so on — don’t exist yet. Until they do, you’re better off trying to make something completely different.


Debugging

Writing an extension for Google Chrome is not always an easy task and you’ll likely run into some problems. The good thing is that you can still use the console to output your variables to help with debugging. Feel free to add console.log into your background or content scripts. However this will not work in scripts which are run in the context of the developer tools, in that case you might consider using the alert method, as it works everywhere.


Conclusion

In my opinion, Chrome is one of the best browsers available. The developers at Google make creating extensions relatively easy by giving us the power to create them in HTML, CSS, and JavaScript.

Yes, there are some tricky parts, but generally we’re able to produce valuable plugins. Keep in mind that this article doesn’t cover everything related to developing Chrome extensions. There are some other useful things like context menus, options pages, and notifications. For the topics I did not cover, please refer to the documentation for more detailed information.

June 20 2013

20:44

Developing With Sass and Chrome DevTools

In this article we’ll take a look at how we can better integrate Chrome DevTools into our Sass development workflow.


Tutorial Sneak Peek!

Here’s a peek of what we’ll be doing in this tutorial via an animated GIF:

edit-sass

On the left is a terminal and the Sublime text editor, and on the right is Chrome. Changes to the .scss file are being persisted to disk, which is reflected in Sublime, the Sass watch command is generating a new version of a CSS file when it detects changes to the Sass file, which is in turn picked up by DevTools and reflected onto the page.


The Need for a Better Workflow

So what are we trying to solve here? Why would we want a better workflow? Well, let’s review an example of debugging some Sass-generated CSS, given the following .scss file, which contains:

$myColor: green;

body {
	.box {
		color: $myColor;
	}
}

And now the CSS, which is generated by the above Sass file, would look like this:

body .box {
  color: green; }

Now we just need a little bit of HTML to apply our styling to:

<div class="box">Hello</div>

Ok, so far so good. Now we want to start designing and developing our page using Chrome DevTools. First we inspect the div with a class of box by right clicking on it and selecting Inspect Element. Then we can check out the styles pane, which shows a green value for the color property, a note on the filename it’s contained within, along with the line number that the selector appears on. Now here’s where the problem lies – the processed CSS does not follow, line for line, what the source .scss file looks like, which can have a negative impact on our development workflow.

sublime-source-vs-output

In the image above, the left pane shows a source .scss file and the right side shows the processed version. (Note I’m using the SCSS bundle for syntax highlighting in Sublime Text)

dev-tools-regular-view

DevTools then provides a link to the file and line number of the selector, highlighted in red above, which unsurprisingly links to the processed file (image below). This now acts as a limitation for editing the file within DevTools as we can’t benefit from file saving. For example, if we persist changes to the processed file, they’ll just be overwritten when Sass next runs.

In fact, we don’t even have a clear indication of what the corresponding line number is – of course, practically speaking, our above example is very simple and we can very quickly understand what line maps to what, by looking at the code. However in a larger .scss file, things can get trickier.

sources-panel-regular

As you can see above, DevTools has no understanding of our .scss file, we can address this however by utilizing a DevTools and Sass workflow, which is what we’ll be looking at in this tutorial. To summarize, a lack of Sass support in the DevTools could mean:

  • We don’t immediately know the line number a selector and/or CSS property exists on, within our source file.
  • Persisting our CSS changes to the file system is pretty much out of the question.
  • Even simply viewing the .scss file in the sources panel is difficult as DevTools doesn’t recognize the extension as one which should be viewed.

Preparing for Sass Support

Chrome

Important: Sass support is an experimental feature. Please keep in mind that, while it has been around for a while now, things may change. If they do, we’ll do our best to update this article, accordingly.

Now let’s navigate to about:flags in the omnibox, then find Enable Developer Tools experiments, enable it and restart Chrome.

chrome-flags

The image below shows an experiments panel, you can reach this by clicking on the cogwheel at the bottom right corner of DevTools and clicking ‘Experiments‘, then check the box for Support for Sass.

dev-tools-experiments

After closing and opening DevTools, you can now go to the General menu item in the left sidebar of the settings window and navigate to the Sources section, from there, check ‘Enable source maps‘ and ‘Auto-reload CSS upon Sass save‘. The timeout can be left at the default value.

sass-support-dev-tools

Sass

This tutorial uses the latest version of Sass (3.3.0 at the current time of this writing) as there is some initial support for source maps. Here’s how you can install Sass using Rubygems:

gem install sass --pre
sass-gem

We now need to tell Sass to compile our .scss files, using the following command:

sass --watch --scss --sourcemap styles.scss:styles.css

From above, the styles.scss is your source file and the styles.css is the processed file, which your HTML page would use. Here’s how you would link in the processed .css file:

<link rel="stylesheet" type="text/css" href="styles.css">

If you see something similar to the image below in your terminal:

sass-command-line

…and you see a generated .map file, then, congratulations: Sass debugging is working! Here’s what those other command options are doing from the previous compile command:

  • --watch: Will watch for changes to the source .scss file and compile it as soon as it has detected a change.
  • --scss: Will display what style of Sass we are using.
  • --sourcemap: Will generate corresponding source map files.

Now, having an understanding of source maps isn’t critical to follow along, however it’s certainly something worth reading about as it’s intended to help the development process. See below for a few resources on source maps:


From Elements to Sources

elem-to-source

Assuming all went well, opening up your page where Sass support is enabled in DevTools should allow for Sass debugging. The first thing to note is that the filename will show the corresponding .scss source file, which is useful, in addition to that, the line number correctly reflects the line number in our source file.

elements-sass

Clicking on the filename will take you right to the sources panel with the line corresponding to the selector in question highlighted.

Note: You now have a viewable Sass file right in your browser with syntax highlighting!

sources-panel-sass

Ok, so finding the corresponding CSS selector from the Elements panel is cool, but what about CSS properties?

In the screenshot below, the Sources panel has highlighted the CSS property I was interested in, to get there I had Command-Clicked on the property from within the Elements panel. The fact that it has taken me to the line where the value was defined (rather than the line: property: $value;) is very powerful!

sources-property-sass

Note: Navigating to a CSS property in the sources panel from command clicking the property in the Elements panel is not specific to Sass, it can be achieved with regular CSS too.


Sass Editing With Persistent Changes

Editing a Sass file is not much different than editing a regular CSS file, one thing we will need to do however is ensure DevTools knows where the .scss file is located at on the file system. From within the Sources panel, the left hand sidebar should have a Sources pane which displays the .scss file, right click it and choose Save as, then overwrite the existing .scss source file.

sources-save-as-sass

Because we checked ‘Auto-reload CSS upon Sass save‘ and because Sass is running in the terminal with the watch flag, changes are made pretty quickly and the DevTools are able to show us those changes.

Even mixins work as expected, pasting the following, in an almost empty .scss file within the sources panel results in the correct CSS being generated, which DevTools is able to reload from within a second.

@mixin button {
    border: 2px solid green;
    display: inline-block;
    padding: 10px;
    border-radius: 4px;    
}

.box {
    @include button;   
}
mixin-sources

As you can see in the image above, the left hand side shows the .scss source file, which is what we would make changes to, the right hand side shows the generated .css file which DevTools has picked up changes from (almost immediately thanks to the watch flag in the terminal)


Sass Debugging at Fluent Conf

Paul Irish gave a great talk at Fluent which had lots of juicy information on the Chrome DevTools, but also the use of Sass. Skip ahead to 5:30 for the Sass information.


Summary

Let’s go over some of the main points discussed in this tutorial:

  • Sass capabilities in Chrome DevTools are a great way to view and modify Sass resources, although don’t forget to enable it in both about:flags and in the settings panel.
  • When using Sass and the Chrome DevTools in a project, you’re likely to have a more efficient workflow with viewing/editing the source file, rather than the generated file.
  • CSS properties in the elements panel can be Command-clicked, the file name displayed to the right of the selector is also clickable. Both will take you to the relevant portion in the sources panel.
  • Utilize the Sources panel’s editing capabilities by doing a ‘Save as‘ on the .scss file, future saves (Command+S) should overwrite the file.
  • Chrome Canary and the latest version of Sass 3.3.0 (due to Source maps) were used in this tutorial, be sure to try them out!

Interesting links

As you can see, the development workflow with Sass is not perfect, but it’s definitely a lot better than what it used to be. As time goes on, DevTools will provide a more streamlined workflow for editing CSS files and bugs will be fixed! Any issues you do find can be raised as a bug at crbug.com.

Thanks for reading this article!

April 29 2013

15:53

HTTP: The Protocol Every Web Developer Must Know – Part 2

In my previous article, we covered some of HTTP’s basics, such as the URL scheme, status codes and request/response headers. With that as our foundation, we will look at the finer aspects of HTTP, like connection handling, authentication and HTTP caching. These topics are fairly extensive, but we’ll cover the most important bits.


HTTP Connections

A connection must be established between the client and server before they can communicate with each other, and HTTP uses the reliable TCP transport protocol to make this connection. By default, web traffic uses TCP port 80. A TCP stream is broken into IP packets, and it ensures that those packets always arrive in the correct order without fail. HTTP is an application layer protocol over TCP, which is over IP.

HTTPS is a secure version of HTTP, inserting an additional layer between HTTP and TCP called TLS or SSL (Transport Layer Security or Secure Sockets Layer, respectively). HTTPS communicates over port 443 by default, and we will look at HTTPS later in this article.

HTTP and HTTPS layers

An HTTP connection is identified by <source-IP, source-port> and <destination-IP, destination-port>. On a client, an HTTP application is identified by a <IP, port> tuple. Establishing a connection between two endpoints is a multi-step process and involves the following:

Connection Delays
  • resolve IP address from host name via DNS
  • establish a connection with the server
  • send a request
  • wait for a response
  • close connection

The server is responsible for always responding with the correct headers and responses.

In HTTP/1.0, all connections were closed after a single transaction. So, if a client wanted to request three separate images from the same server, it made three separate connections to the remote host. As you can see from the above diagram, this can introduce lot of network delays, resulting in a sub-optimal user experience.

To reduce connection-establishment delays, HTTP/1.1 introduced persistent connections, long-lived connections that stay open until the client closes them. Persistent connections are default in HTTP/1.1, and making a single transaction connection requires the client to set the Connection: close request header. This tells the server to close the connection after sending the response.

In addition to persistent connections, browsers/clients also employ a technique, called parallel connections, to minimize network delays. The age-old concept of parallel connections involves creating a pool of connections (generally capped at six connections). If there are six assets that the client needs to download from a website, the client makes six parallel connections to download those assets, resulting in a faster turnaround. This is a huge improvement over serial connections where the client only downloads an asset after completing the download for a previous asset.

Parallel connections, in combination with persistent connections, is today’s answer to minimizing network delays and creating a smooth experience on the client. For an in-depth treatment of HTTP connections, refer to the Connections section of the HTTP spec.

Server-side Connection Handling

The server mostly listens for incoming connections and processes them when it receives a request. The operations involve:

  • establishing a socket to start listening on port 80 (or some other port)
  • receiving the request and parsing the message
  • processing the response
  • setting response headers
  • sending the response to the client
  • close the connection if a Connection: close request header was found

Of course, this is not an exhaustive list of operations. Most applications/websites need to know who makes a request in order to create more customized responses. This is the realm of identification and authentication.


Identification and Authentication

HTTP is an application layer protocol over TCP, which is over IP.

It is almost mandatory to know who connects to a server for tracking an app’s or site’s usage and the general interaction patterns of users. The premise of identification is to tailor the response in order to provide a personalized experience; naturally, the server must know who a user is in order to provide that functionality.

There are a few different ways a server can collect this information, and most websites use a hybrid of these approaches:

  • Request headers: From, Referer, User-Agent – We saw these headers in Part 1.
  • Client-IP – the IP address of the client
  • Fat Urls – storing state of the current user by modifying the URL and redirecting to a different URL on each click; each click essentially accumulates state.
  • Cookies – the most popular and non-intrusive approach.

Cookies allow the server to attach arbitrary information for outgoing responses via the Set-Cookie response header. A cookie is set with one or more name=value pairs separated by semicolon (;), as in Set-Cookie: session-id=12345ABC; username=nettuts.

A server can also restrict the cookies to a specific domain and path, and it can make them persistent with an expires value. Cookies are automatically sent by the browser for each request made to a server, and the browser ensures that only the domain- and path-specific cookies are sent in the request. The request header Cookie: name=value [; name2=value2] is used to send these cookies to the server.

The best way to identify a user is to require them to sign up and log in, but implementing this feature requires some effort by the developer, as well as the user.

Techniques like OAuth simplify this type of feature, but it still requires user consent in order to work properly. Authentication plays a large role here, and it is probably the only way to identify and verify the user.

Authentication

HTTP does support a rudimentary form of authentication called Basic Authentication, as well as the more secure Digest Authentication.

In Basic Authentication, the server initially denies the client’s request with a WWW-Authenticate response header and a 401 Unauthorized status code. On seeing this header, the browser displays a login dialog, prompting for a username and password. This information is sent in a base-64 encoded format in the Authentication request header. The server can now validate the request and allow access if the credentials are valid. Some servers might also send an Authentication-Info header containing additional authentication details.

Authentication Challenge/Response

A corollary to Basic-Authentication is Proxy Authentication. Instead of a web server, the authetication challenge is requested by an intermediate proxy. The proxy sends a Proxy-Authenticate header with a 407 Unauthorized status code. In return, the client is supposed to send the credentials via the Proxy-Authorization request header.

Digest Authentication is similar to Basic and uses the same handshake technique with the WWW-Authenticate and Authorization headers, but Digest uses a more secure hashing function to encrypt the username and password (commonly with MD5 or KD digest functions). Although Digest Authentication is supposed to be more secure than Basic, websites typically use Basic Authentication because of its simplicty. To mitigate the security concerns, Basic Auth is used in conjunction with SSL.

Secure HTTP

HTTPS AddressBar

The HTTPS protocol provides a secure connection on the web. The easiest way to know if you are using HTTPS is to check the browser’s address bar. HTTPs’ secure component involves inserting a layer of encryption/decryption between HTTP and TCP. This is the Secure Sockets Layer (SSL) or the improved Transport Layer Security (TLS).

SSL uses a powerful form of encryption using RSA and public-key cryptography. Because secure transactions are so important on the web, a ubiquitous standards-based Public-Key Infrastructure (PKI) effort has been underway for quite sometime.

Existing clients/servers do not have to change the way they handle messages because most of the hard work happens in the SSL layer. Thus, you can develop your web application using Basic Authentication and automatially reap the benefits of SSL by switching to the https:// protocol. However, to make the web application work over HTTPS, you need to have a working digital certificate deployed on the server.

Certificates

Just as you need ID cards to show your identity, a web server needs a digital certificate to identify itself. Certificates (or “certs”) are issued by a Certificate Authority (CA) and vouch for your identity on the web. The CAs are the guardians of the PKI. The most common form of certificates is the X.509 v3 standard, which contains information, such as:

  • the certificate issuer
  • the algorithm used for the certificate
  • the subject name or organization for whom this cert is created
  • the public key information for the subject
  • the Certification Authority Signature, using the specified signing algorithm

When a client makes a request over HTTPS, it first tries to locate a certificate on the server. If the cert is found, it attempts to verfiy it against its known list of CAs. If its not one of the listed CAs, it might show a dialog to the user warning about the website’s certficate.

Once the certificate is verified, the SSL handshake is complete and secure transmission is in effect.


HTTP Caching

It is generally agreed that doing the same work twice is wasteful. This is the guiding principle around the concept of HTTP caching, a fundamental pillar of the HTTP Network Infrastructure. Because most of the operations are over a network, a cache helps save time, cost and bandwidth, as well as provide an improved experience on the web.

Caches are used at several places in the network infrastructure, from the browser to the origin server. Depending on where it is located, a cache can be categorized as:

  • Private: within a browser, caches usernames, passwords, URLs, browsing history and web content. They are generally small and specific to a user.
  • Public: deployed as caching proxies between the server and client. These are much larger because they serve multiple users. A common practice is to keep multiple caching proxies between the client and the origin-server. This helps to serve frequently accessed content, while still allowing a trip to the server for infrequently needed content.
Cache Topology

Cache Processing

Regardless of where a cache is located, the process of maintaining a cache is quite similar:

  • Receive request message.
  • Parse the URL and headers.
  • Lookup a local copy; otherwise, fetch and store locally
  • Do a freshness check to determine the age of the content in the cache; make a request to refresh the content only if necessary.
  • Create the response from the cached body and updated headers.
  • Send the response back to client.
  • Optionally, log the transaction.

Of course, the server is responsible for always responding with the correct headers and responses. If a document hasn’t changed, the server should respond with a 304 Not Modified. If the cached copy has expired, it should generate a new response with updated response headers and return with a 200 OK. If the resource is deleted, it should come back with 404 Not Found. These responses help tune the cache and ensure that stale content is not kept for too long.

Cache Control Headers

Parallel connections, in combination with persistent connections, is today’s answer to minimizing network delays.

Now that we have a sense of how a cache works, it’s time to look at the request and response headers that enable the caching infrastructure. Keeping the content fresh and up-to-date is one of the primary responsibilities of the cache. To keep the cached copy consistent with the server, HTTP provides some simple mechanisms, namely Document Expiration and Server Revalidation.

Document Expiration

HTTP allows an origin-server to attach an expiration date to each document using the Cache-Control and Expires response headers. This helps the client and other cache servers know how long a document is valid and fresh. The cache can serve the copy as long as the age of the document is within the expiration date. Once a document expires, the cache must check with the server for a newer copy and update its local copy accordingly.

Expires is an older HTTP/1.0 response header that specifies the value as an absolute date. This is only useful if the server clocks are in sync with the client, which is a terrible assumption to make. This header is less useful compared to the newer Cache-Control: max-age=<s> header introduced in HTTP/1.1. Here, max-age is a relative age, specified in seconds, from the time the response was created. Thus if a document should expire after one day, the expiration header should be Cache-Control: max-age=86400.

Server Revalidation

Once a cached document expires, the cache must revalidate with the server to check if the document has changed. This is called server revalidation and serves as a querying mechanism for the stale-ness of a document. Just because a cached copy has expired doesn’t mean that the server actually has newer content. Revalidation is just a means of ensuring that the cache stays fresh. Because of the expiration time (as specified in a previous server response), the cache doesn’t have to check with the server for every single request, thus saving bandwidth, time and reducing the network traffic.

The combination of document expiration and server revalidation is a very effective mechanism, it and allows distributed systems to maintain copies with an expiration date.

If the content is known to frequently change, the expiration time can be reduced—allowing the systems to re-sync more frequently.

The revalidation step can be accomplished with two kinds of request-headers: If-Modified-Since and If-None-Match. The former is for date-based validation while the latter uses Entity-Tags (ETags), a hash of the content. These headers use date or ETag values obtained from a previous server response. In case of If-Modified-Since, the Last-Modified response header is used; for If-None-Match, it is the ETag response header.

Controlling the Cachability

The validity period for a document should be defined by the server generating the document. If it’s a newspaper website, the homepage should expire after a day (or sometimes even every hour!). HTTP provides the Cache-Control and Expires response headers to set the expiration on documents. As mentioned earlier, Expires is based on absolute dates and not a reliable solution for controlling cache.

The Cache-Control header is far more useful and has a few different values to constrain how clients should be caching the response:

  • Cache-Control: no-cache: the client is allowed to store the document; however, it must revalidate with the server on every request. There is a HTTP/1.0 compatibility header called Pragma: no-cache, which works the same way.
  • Cache-Control: no-store: this is a stronger directive to the client to not store the document at all.
  • Cache-Control: must-revalidate: this tells the client to bypass its freshness calculation and always revalidate with the server. It is not allowed to serve the cached response in case the server is unavailable.
  • Cache-Control: max-age: this sets a relative expiration time (in seconds) from the time the response is generated.

As an aside, if the server does not send any Cache-Control headers, the client is free to use its own heuristic expiration algorithm to determine freshness.

Constraining Freshness from the Client

Cachability is not just limited to the server. It can also be specified from the client. This allows the client to impose constraints on what it is willing to accept. This is possible via the same Cache-Control header, albeit with a few different values:

  • Cache-Control: min-fresh=<s>: the document must be fresh for at least <s> seconds.
  • Cache-Control: max-stale or Cache-Control: max-stale=<s>: the document cannot be served from the cache if it has been stale for longer than <s> seconds.
  • Cache-Control: max-age=<s>: the cache cannot return a document that has been cached longer than <s> seconds.
  • Cache-Control: no-cache or Pragma: no-cache: the client will not accept a cached resource unless it has been revalidated.

HTTP Caching is actually a very interesting topic, and there are some very sophisticated algorithms to manage cached content. For a deeper look into this topic, refer to the Caching section of the HTTP spec.


Summary

Our tour of HTTP began with the foundation of URL schemes, status codes and request/response headers. Building upon those concepts, we looked at some of the finer areas of HTTP, such as connection handling, identification and authentication and caching. I am hopeful that this tour has given you a good taste for the breadth of HTTP and enough pointers to further explore this protocol.

References

April 26 2013

17:52

Test-Driving Shell Scripts

Writing shell scripts is very much like programming. Some scripts require little time investment; whereas, other complex scripts may require thought, planning and a larger commitment. From this perspective, it makes sense to take a test-driven approach and unit test our shell scripts.

To get the most out of this tutorial, you need to be familiar with the command line interface (CLI); you may want to check out the The Command Line is Your Best Friend tutorial if you need a refresher. You also need a basic understanding of Bash-like shell scripting. Finally, you may want to familiarize yourself with the test-driven development (TDD) concepts and unit testing in general; be sure to check out these Test-Driven PHP tutorials to get the basic idea.


Prepare the Programming Environment

First, you need a text editor to write your shell scripts and unit tests. Use your favorite!

We will use the shUnit2 shell unit testing framework to run our unit tests. It was designed for, and works with, Bash-like shells. shUnit2 is an open source framework released under the GPL license, and a copy of the framework is also included with this tutorial’s sample source code.

Installing shUnit2 is very easy; simply download and extract the archive to any location on your hard drive. It is written in Bash, and as such, the framework consists of only script files. If you plan to frequently use shUnit2, I highly recommend that you put it in a location in your PATH.


Writing our First Test

For this tutorial, extract shUnit into a directory with the same name in your Sources folder (see the code attached to this tutorial). Create a Tests folder inside Sources and added a new file call firstTest.sh.

#! /usr/bin/env sh

### firstTest.sh ###

function testWeCanWriteTests () {
	assertEquals "it works" "it works"
}

## Call and Run all Tests
. "../shunit2-2.1.6/src/shunit2"

Than make your test file executable.

$ cd __your_code_folder__/Tests
$ chmod +x firstTest.sh

Now you can simply run it and observe the output:

$ ./firstTest.sh
testWeCanWriteTests

Ran 1 test.

OK

It says we ran one successful test. Now, let’s cause the test to fail; change the assertEquals statement so that the two strings are not the same and run the test again:

$ ./firstTest.sh
testWeCanWriteTests
ASSERT:expected:<it works> but was:<it does not work>

Ran 1 test.

FAILED (failures=1)

A Tennis Game

You write acceptance tests at the beginning of a project/feature/story when you can clearly define a specific requirement.

Now that we have a working testing environment, let’s write a script that reads a file, makes decisions based on the file’s contents and outputs information to the screen.

The main goal of the script is to show the score of a tennis game between two players. We will concentrate only on keeping the score of a single game; everything else is up to you. The scoring rules are:

  • At the beginning, each player has a score of zero, called “love”
  • First, second and third balls won are marked as “fifteen”, “thirty”, and “forty”.
  • If at “forty” the score is equal, it is called “deuce”.
  • After this, the score is kept as “Advantage” for the player who scores one more point than the other player.
  • A player is the winner if he manages to have an advantage of at least two points and wins at least three points (that is, if he reached at least “forty”).

Definition of Input and Output

Our application will read the score from a file. Another system will push the information into this file. The first line of this data file will contain the names of the players. When a player scores a point, their name is written at the end of the file. A typical score file looks like this:

John - Michael
John
John
Michael
John
Michael
Michael
John
John

You can find this content in the input.txt file in the Source folder.

The output of our program writes the score to the screen one line at a time. The output should be:

John - Michael
John: 15 - Michael: 0
John: 30 - Michael: 0
John: 30 - Michael: 15
John: 40 - Michael: 15
John: 40 - Michael: 30
Deuce
John: Advantage
John: Winner

This output can be also found in the output.txt file. We will use this information to check if our program is correct.


The Acceptance Test

You write acceptance tests at the beginning of a project/feature/story when you can clearly define a specific requirement. In our case, this test simply calls our soon-to-be-created script with the name of the input file as the parameter, and it expects the output to be identical with the hand-written file from the previous section:

#! /usr/bin/env sh

### acceptanceTest.sh ###

function testItCanProvideAllTheScores () {
	cd ..
	./tennisGame.sh ./input.txt > ./results.txt
	diff ./output.txt ./results.txt
	assertTrue 'Expected output differs.' $?
}

## Call and Run all Tests
. "../shunit2-2.1.6/src/shunit2"

We will run our tests in the Source/Tests folder; therefore, cd .. takes us into the Source directory. Then it tries to run tennisGamse.sh, which does not yet exist. Then the diff command will compare the two files: ./output.txt is our hand-written output and ./results.txt will contain the result of our script. Finally, assertTrue checks the exit value of diff.

But for now, our test returns the following error:

$ ./acceptanceTest.sh
testItCanProvideAllTheScores
./acceptanceTest.sh: line 7: tennisGame.sh: command not found
diff: ./results.txt: No such file or directory
ASSERT:Expected output differs.

Ran 1 test.

FAILED (failures=1)

Let’s turn those errors into a nice failure by creating an empty file called tennisGame.sh and make it executable. Now when we run our test, we don’t get an error:

./acceptanceTest.sh
testItCanProvideAllTheScores
1,9d0
< John - Michael
< John: 15 - Michael: 0
< John: 30 - Michael: 0
< John: 30 - Michael: 15
< John: 40 - Michael: 15
< John: 40 - Michael: 30
< Deuce
< John: Advantage
< John: Winner
ASSERT:Expected output differs.

Ran 1 test.

FAILED (failures=1)

Implementation with TDD

Create another file called unitTests.sh for our unit tests. We don’t want to run our script for each test; we only want to run the functions that we test. So, we will make tennisGame.sh run only the functions that will reside in functions.sh:

#! /usr/bin/env sh

### unitTest.sh ###

source ../functions.sh

function testItCanProvideFirstPlayersName () {
	assertEquals 'John' `getFirstPlayerFrom 'John - Michael'`
}

## Call and Run all Tests
. "../shunit2-2.1.6/src/shunit2"

Our first test is simple. We attempt to retrieve the first player’s name when a line contains two names separated by a hyphen. This test will fail because we do not yet have a getFirstPlayerFrom function:

$ ./unitTest.sh
testItCanProvideFirstPlayersName
./unitTest.sh: line 8: getFirstPlayerFrom: command not found
shunit2:ERROR assertEquals() requires two or three arguments; 1 given
shunit2:ERROR 1: John 2:  3:

Ran 1 test.

OK

The implementation for getFirstPlayerFromis very simple. It’s a regular expression that is pushed through the sed command:

### functions.sh ###

function getFirstPlayerFrom () {
	echo $1 | sed -e 's/-.*//'
}

Now the test passes:

$ ./unitTest.sh
testItCanProvideFirstPlayersName

Ran 1 test.

OK

Let’s write another test for the second player’s name:

### unitTest.sh ###

[...]

function testItCanProvideSecondPlayersName () {
	assertEquals 'Michael' `getSecondPlayerFrom 'John - Michael'`
}

The failure:

./unitTest.sh
testItCanProvideFirstPlayersName
testItCanProvideSecondPlayersName
ASSERT:expected:<Michael> but was:<John>

Ran 2 tests.

FAILED (failures=1)

And now the function implementation to make it pass:

### functions.sh ###

[...]

function getSecondPlayerFrom () {
	echo $1 | sed -e 's/.*-//'
}

Now we have passing tests:

$ ./unitTest.sh
testItCanProvideFirstPlayersName
testItCanProvideSecondPlayersName

Ran 2 tests.

OK

Let’s Speed Things Up

Starting at this point, we will write a test and the implementation, and I will explain only what deserves to be mentioned.

Let’s test if we have a player with only one score. Added the following test:

function testItCanGetScoreForAPlayerWithOnlyOneWin () {
	standings=$'John - Michael
John'
	assertEquals '1' `getScoreFor 'John' "$standings"`
}

And the solution:

function getScoreFor () {
	player=$1
	standings=$2
	totalMatches=$(echo "$standings" | grep $player | wc -l)
	echo $(($totalMatches-1))
}

We use some fancy-pants quoting to pass the newline sequence ( ) inside a string parameter. Then we use grep to find the lines that contain the player’s name and count them with wc. Finally, we subtract one from the result to counteract the presence of the first line (it contains only non-score related data).

Now we are at the refactoring phase of TDD.

I just realized that the code actually works for more than one point per player, and we can refactor our tests to reflect this. Change the above test function to the following:

function testItCanGetScoreForAPlayer () {
	standings=$'John - Michael
John
Michael
John'
	assertEquals '2' `getScoreFor 'John' "$standings"`
}

The tests still passes. Time to move on with our logic:

function testItCanOutputScoreAsInTennisForFirstPoint () {
	assertEquals 'John: 15 - Michael: 0' "`displayScore 'John' 1 'Michael' 0`"
}

And the implementation:

function displayScore () {
	if [ "$2" -eq '1' ]; then
		playerOneScore='15'
	fi

	echo "$1: $playerOneScore - $3: $4"
}

I only check the second parameter. This looks like I’m cheating, but it is the simplest code to make the test pass. Writing another test forces us to add more logic, but what test should we write next?

There are two paths we can take. Testing if the second player recieves a point forces us to write another if statement, but we only have to add an else statement if we choose to test the first player’s second point. The latter implies an easier implementation, so let’s try that:

function testItCanOutputScoreAsInTennisForSecondPointFirstPlayer () {
	assertEquals 'John: 30 - Michael: 0' "`displayScore 'John' 2 'Michael' 0`"
}

And the implementation:

function displayScore () {
	if [ "$2" -eq '1' ]; then
		playerOneScore='15'
	else
		playerOneScore='30'
	fi

	echo "$1: $playerOneScore - $3: $4"
}

This still looks cheating, but it works perfectly. Continuing on for the third point:

function testItCanOutputScoreAsInTennisForTHIRDPointFirstPlayer () {
	assertEquals 'John: 40 - Michael: 0' "`displayScore 'John' 3 'Michael' 0`"
}

The implementation:

function displayScore () {
	if [ "$2" -eq '1' ]; then
		playerOneScore='15'
	elif [ "$2" -eq '2' ]; then
		playerOneScore='30'
	else
		playerOneScore='40'
	fi

	echo "$1: $playerOneScore - $3: $4"
}

This if-elif-else is starting to annoy me. I want to change it, but let’s first refactor our tests. We have three very similar tests; so let’s write them into a single test that makes three assertions:

function testItCanOutputScoreWhenFirstPlayerWinsFirst3Points () {
	assertEquals 'John: 15 - Michael: 0' "`displayScore 'John' 1 'Michael' 0`"
	assertEquals 'John: 30 - Michael: 0' "`displayScore 'John' 2 'Michael' 0`"
	assertEquals 'John: 40 - Michael: 0' "`displayScore 'John' 3 'Michael' 0`"
}

That’s better, and it still passes. Now, let’s create a similar test for the second player:

function testItCanOutputScoreWhenSecondPlayerWinsFirst3Points () {
	assertEquals 'John: 0 - Michael: 15' "`displayScore 'John' 0 'Michael' 1`"
	assertEquals 'John: 0 - Michael: 30' "`displayScore 'John' 0 'Michael' 2`"
	assertEquals 'John: 0 - Michael: 40' "`displayScore 'John' 0 'Michael' 3`"
}

Running this test results in interesting output:

testItCanOutputScoreWhenSecondPlayerWinsFirst3Points
ASSERT:expected:<John: 0 - Michael: 15> but was:<John: 40 - Michael: 1>
ASSERT:expected:<John: 0 - Michael: 30> but was:<John: 40 - Michael: 2>
ASSERT:expected:<John: 0 - Michael: 40> but was:<John: 40 - Michael: 3>

Well that was unexpected. We knew that Michael would have incorrect scores. The surprise is John; he should have 0 not 40. Let’s fix that by first modifying the if-elif-else expression:

function displayScore () {
	if [ "$2" -eq '1' ]; then
		playerOneScore='15'
	elif [ "$2" -eq '2' ]; then
		playerOneScore='30'
	elif [ "$2" -eq '3' ]; then
		playerOneScore='40'
	else
		playerOneScore=$2
	fi

	echo "$1: $playerOneScore - $3: $4"
}

The if-elif-else is now more complex, but we at least fixed the John’s scores:

testItCanOutputScoreWhenSecondPlayerWinsFirst3Points
ASSERT:expected:<John: 0 - Michael: 15> but was:<John: 0 - Michael: 1>
ASSERT:expected:<John: 0 - Michael: 30> but was:<John: 0 - Michael: 2>
ASSERT:expected:<John: 0 - Michael: 40> but was:<John: 0 - Michael: 3>

Now let’s fix Michael:

function displayScore () {
	echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
}

function convertToTennisScore () {
	if [ "$1" -eq '1' ]; then
		playerOneScore='15'
	elif [ "$1" -eq '2' ]; then
		playerOneScore='30'
	elif [ "$1" -eq '3' ]; then
		playerOneScore='40'
	else
		playerOneScore=$1
	fi

	echo $playerOneScore;
}

That worked well! Now it’s time to finally refactor that ugly if-elif-else expression:

function convertToTennisScore () {
	declare -a scoreMap=('0' '15' '30' '40')
	echo ${scoreMap[$1]};
}

Value maps are wonderful! Let’s move on to the “Deuce” case:

function testItSayDeuceWhenPlayersAreEqualAndHaveEnoughPoinst () {
	assertEquals 'Deuce' "`displayScore 'John' 3 'Michael' 3`"
}

We check for “Deuce” when all players have at least a score of 40.

function displayScore () {
	if [ $2 -gt 2 ] && [ $4 -gt 2 ] && [ $2 -eq $4 ]; then
		echo "Deuce"
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

Now we test for the first player’s advantage:

function testItCanOutputAdvantageForFirstPlayer () {
	assertEquals 'John: Advantage' "`displayScore 'John' 4 'Michael' 3`"
}

And to make it pass:

function displayScore () {
	if [ $2 -gt 2 ] && [ $4 -gt 2 ] && [ $2 -eq $4 ]; then
		echo "Deuce"
	elif [ $2 -gt 2 ] && [ $4 -gt 2 ] && [ $2 -gt $4 ]; then
		echo "$1: Advantage"
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

There’s that ugly if-elif-else again, and we have a lot of duplication as well. All our tests pass, so let’s refactor:

function displayScore () {
	if outOfRegularScore $2 $4 ; then
		checkEquality $2 $4
		checkFirstPlayerAdv $1 $2 $4
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

function outOfRegularScore () {
	[ $1 -gt 2 ] && [ $2 -gt 2 ]
	return $?
}

function checkEquality () {
	if [ $1 -eq $2 ]; then
		echo "Deuce"
	fi
}

function checkFirstPlayerAdv () {
	if [ $2 -gt $3 ]; then
		echo "$1: Advantage"
	fi
}

This’ll work for now. Let’s test the advantage for the second player:

function testItCanOutputAdvantageForSecondPlayer () {
	assertEquals 'Michael: Advantage' "`displayScore 'John' 3 'Michael' 4`"
}

And the code:

function displayScore () {
	if outOfRegularScore $2 $4 ; then
		checkEquality $2 $4
		checkAdvantage $1 $2 $3 $4
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

function checkAdvantage () {
	if [ $2 -gt $4 ]; then
		echo "$1: Advantage"
	elif [ $4 -gt $2 ]; then
		echo "$3: Advantage"
	fi
}

This works, but we have some duplication in the checkAdvantage function. Let’s simplify it and call it twice:

function displayScore () {
	if outOfRegularScore $2 $4 ; then
		checkEquality $2 $4
		checkAdvantage $1 $2 $4
		checkAdvantage $3 $4 $2
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

function checkAdvantage () {
	if [ $2 -gt $3 ]; then
		echo "$1: Advantage"
	fi
}

This is actually better than our previous solution, and it reverts to the original implementation of this method. But we now we have another problem: I feel uncomfortable with the $1, $2, $3 and $4 variables. They need meaningful names:

function displayScore () {
	firstPlayerName=$1; firstPlayerScore=$2
	secondPlayerName=$3; secondPlayerScore=$4

	if outOfRegularScore $firstPlayerScore $secondPlayerScore; then
		checkEquality $firstPlayerScore $secondPlayerScore
		checkAdvantageFor $firstPlayerName $firstPlayerScore $secondPlayerScore
		checkAdvantageFor $secondPlayerName $secondPlayerScore $firstPlayerScore
	else
		echo "$1: `convertToTennisScore $2` - $3: `convertToTennisScore $4`"
	fi
}

function checkAdvantageFor () {
	if [ $2 -gt $3 ]; then
		echo "$1: Advantage"
	fi
}

This makes our code longer, but it is significantly more expressive. I like it.

It’s time to find a winner:

function testItCanOutputWinnerForFirstPlayer () {
	assertEquals 'John: Winner' "`displayScore 'John' 5 'Michael' 3`"
}

We only have to modify the checkAdvantageFor function:

function checkAdvantageFor () {
	if [ $2 -gt $3 ]; then
		if [ `expr $2 - $3` -gt 1 ]; then
			echo "$1: Winner"
		else
			echo "$1: Advantage"
		fi
	fi
}

We are almost done! As our last step, we’ll write the code in tennisGame.sh to make the acceptance test pass. This will be fairly simple code:

#! /usr/bin/env sh

### tennisGame.sh ###

. ./functions.sh

playersLine=`head -n 1 $1`
echo "$playersLine"
firstPlayer=`getFirstPlayerFrom "$playersLine"`
secondPlayer=`getSecondPlayerFrom "$playersLine"`

wholeScoreFileContent=`cat $1`
totalNoOfLines=`echo "$wholeScoreFileContent" | wc -l`
for currentLine in `seq 2 $totalNoOfLines`
	do
	firstPlayerScore=$(getScoreFor $firstPlayer "`echo "$wholeScoreFileContent" | head -n $currentLine`")
	secondPlayerScore=$(getScoreFor $secondPlayer "`echo "$wholeScoreFileContent" | head -n $currentLine`")
	displayScore $firstPlayer $firstPlayerScore $secondPlayer $secondPlayerScore
done

We read the first line to retrieve the names of the two players, and then we incrementally read the file to compute the score.


Final Thoughts

Shell scripts can easily grow from a few lines of code to a few hundred of lines. When this happens, maintenance becomes increasingly difficult. Using TDD and unit testing can greatly help to make your complex script easier to maintain—not to mention that it forces you to build your complex scripts in a more professional manner.

April 19 2013

21:27

Writing a Shell Script From Scratch

Writing shell scripts can be rather daunting, primarily because the shell isn't the most friendly of languages to use. However, I hope to show you in this tutorial that shell scripting is actually not as tough or as scary as you might expect.

For this tutorial, we'll write a script that makes the process of using the Jasmine test framework a little bit easier. Actually, I wouldn't use this script today; I would use Grunt.js or something similar. However, I wrote this script before Grunt was around, and I found that writing it proved to be an excellent way to get more comfortable with shell scripting, so that's why we're using it.

One note: this tutorial is loosely associated with my upcoming Tuts+ Premium course, "Advanced Command Line Techniques." To learn more about pretty much anything in this tutorial, stay tuned for that course’s release. Hereafter in this tutorial, it will be referred to as "the course."

So, our script, which I call jazz, will have four main features:

  • It will download Jasmine from the web, unzip it, and delete the sample code.
  • It will create JavaScript files and their associated spec files, and pre-fill them with a bit of template code.
  • It will open the tests in the browser.
  • It will display the help text, which outlines the above.

Let's begin with the script file.


Step 1 - Creating the File

Writing a shell script is only useful if you can use it from the terminal; to be able to use your custom scripts on the terminal, you need to put them in a folder that is in your terminal's PATH variable (you can see your PATH variable by running echo $PATH). I've created a ~/bin folder (where ~ is the home directory) on my computer, and that's where I like to keep custom scripts (if you do the same, you'll have to add it to your path). So, just create a file, called jazz, and put it in your folder.

Of course, we'll also have to make that file executable; otherwise, we won't be able to run it. We can do this by running the following command:

    chmod +x jazz

Now that we can actually execute the script, let's add a very important part. All shell scripts should begin with a shebang). As Wikipedia says, this should be the first line of the script; it states what interpreter, or shell, this script should be run with. We're just going to use a basic, standard shell:

    #!/bin/sh

All right, with all that set up, we're ready to start writing the actual code.


Step 2 – Outlining the Script Flow

Earlier, I pointed out what the different features of our shell script should be. But how will the script know which feature to run? We'll use a combination of a shell parameter and a case statement. When running the script from the command line, we'll use a sub-command, like this:

    jazz init
    jazz create SomeFile
    jazz run
    jazz help

This should look familiar, especially if you've used Git:

    git init
    git status
    git commit

Based on that first parameter (init, create, run, help), our case statement will decide what to run. However, we do need a default case: what happens if no first parameter is given, or we get an unrecognized first parameter? In those cases, we'll show the help text. So, let's get started!


Step 3 – Writing the Help Text

We begin with the if statement that checks for our first parameter:

    if [ $1 ]
    then
        # do stuff 
    else
        # show help
    fi

You might be a little confused at first, because a shell if statement is pretty different from a "regular" programming language's if statement. To get a better understanding of it, watch the screencast on conditional statements in the course. This code checks for the presence of a first parameter ($1); if it's there, we'll execute the then code; else, we'll show the help text.

It's a good idea to wrap the printing of the help text in a function, because we need to call it more than once. We do need to define the function before it is called, so we'll put it at the top. I like this, because now, as soon as I open the file, I see the documentation for the script, which can be a helpful reminder when returning to code you haven't seen in a while. Without further ado, here's the help function:

    function help () {
        echo "jazz - A simple script that makes using the Jasmine testing framework in a standalone project a little simpler."
        echo "
        echo "    jazz init                  - include jasmine in the project";
        echo "    jazz create FunctionName   - creates ./src/FunctionName.js ./spec/FunctionNameSpec.js";
        echo "    jazz run                   - runs tests in browser";
    }

Now, just replace that # show help function with a call to the help function.

    else
        help
    fi


Step 4 – Writing the Case Statement

If there is a first parameter, we need to figure out which it is. For this, we use a case statement:

	case "$1" in
		init)
		
		;;
		create)
		
		;;
		run)
		
		;;
		*)
			help
		;;
	esac

We pass the first parameter to the case statement; then, it should match one of four things: "init", "create", "run", or our wildcard, default case. Notice that we don't have an explicit "help" case: that's just our default case. This works, because anything other than "init", "create", and "run" aren't commands we recognize, so it should get the help text.

Now we're ready to write the functional code, and we'll start with jazz init.


Step 5 – Preparing Jasmine with jazz init

All the code we write here will go within our init) case, from the case statement above. The first step is to actually download the standalone version of Jasmine, which comes in a zip file:

    echo "Downloading Jasmine ..."
    curl -sO $JASMINE_LINK

We first echo a little message, and then we use curl to download the zip. The s flag makes it silent (no output) and the O flag saves the contents of the zip to a file (otherwise, it would pipe it to standard out). But what's with that $JASMINE_LINK variable? Well, you could put the actual link to the zip file there, but I prefer to put it in a variable for two reasons: first, it keeps us from repeating part of the path, as you'll see in a minute. Second, with that variable near the top of the file, it makes it easy to change the version of Jasmine we're using: just change that one variable. Here's that variable declaration (I put it outside the if statement, near the top):

    JASMIME_LINK="http://cloud.github.com/downloads/pivotal/jasmine/jasmine-standalone-1.3.1.zip"

Remember, no spaces around the equals-sign in that line.

Now that we have our zip file, we can unzip it and prepare the contents:

    unzip -q <code>basename $JASMINE_LINK</code>
    rm -rf <code>basename $JASMINE_LINK</code> src/*.js spec/*.js

In two of these lines, we're using basename $JASMINE_LINK; the basename command just reduces a path to the base name: so path/to/file.zip becomes just file.zip. This allows us to use that $JASMINE_LINK variable to reference our local zip file.

After we unzip, we'll delete that zip file, as well as all the JavaScript files in the src and spec directories. These are the sample files that Jasmine comes with, and we don't need them.

Next, we have a Mac-only issue to deal with. By default, when you download something from the internet on a Mac, when you try to run it for the first time, you'll be asked to confirm that you want to run it. This is because of the extended attribute com.apple.quarantine that Apple puts on the file. We need to remove this attribute.

    if which xattr > /dev/null && [ "<code>xattr SpecRunner.html</code>" = "com.apple.quarantine" ]
    then
        xattr -d com.apple.quarantine SpecRunner.html
    fi

We begin by checking for the presence of the xattr command, because it doesn't exist on some Unix systems (I'm not sure, but it may be a Mac-only program). If you watched the course screencast on conditionals, you'll know that we can pass any command to if; if it has an exit status of anything but 0, it's false. If which finds the xattr command, it will exit with 0; otherwise, it will exit with 1. In either case, which will display some output; we can keep that from showing by redirecting it to /dev/null (this is a special file that discards all data written to it).

That double ampersand is an boolean AND; it's there for the second condition we want to check. That is, does the SpecRunner.html have that attribute? We can simply run the xattr command on the file, and compare it's output to the string we're expecting. (We can't just expect the file to have the attribute, because you can actually turn this feature off in Mac OS X, and we'll get an error when trying to remove it if the file doesn't have the attribute).

So, if xattr is found and the file has the attribute, we'll remove it, with the d (for delete) flag. Pretty straightforward, right?

The final step is to edit SpecRunner.html. Currently, it contains scripts tags for the sample JavaScript files that we deleted; we should also delete those scripts tags. I happen to know that those script tags span lines 12 to 18 in the files. So, we can use the stream editor sed to delete those lines:

    sed -i "" '12,18d' SpecRunner.html
    echo "Jasmine initialized!"

The i flag tells sed to edit the file in place, or to save the output from the command to the same file we passed in; the empty string after the flag means that we don't want sed to back up the file for us; if you wanted that, you could just put a file extension in that string (like .bak, to get SpecRunner.html.bak).

Finally, we'll let the user know that Jasmine has been initialized. And that's it for our jazz init command.


Step 6 – Creating Files with jazz create

Next up, we're going to let our users create JavaScript files and their associated spec files. This portion of the code will go in the “create" section of the case statement we wrote earlier.

    if [ $2 ]
    then
        # create files
    else
        echo "please include a name for the file"
    fi

When using jazz create, we need to include a name for the file as the second parameter: jazz create View, for example. We'll use this to create src/View.js and spec/ViewSpec.js. So, if there isn't a second parameter, we'll remind the user to add one.

If there is a file name, we'll start by creating those two files (inside the then part above):

    echo "function $2 () {

}" > src/$2.js
    echo "describe('$2', function () {

});" > spec/$2Spec.js

Of course, you can put whatever you want into your src file. I'm doing something basic here; so jazz create View will create src/View.js with this:

function View () {

}

You could replace that first echo line with this:

    echo "var $2 = (function () {
	var $2Prototype = {

	};

	return {
		create : function (attrs) {
			var o = Object.create($2Prototype);
			extend(o, attrs);
			return o;
		}
 	};
}());" > src/$2.js

And then jazz create View will result in this:

    var View = (function () {
        var ViewPrototype = {
        
        };
    
        return {
            create : function (attrs) {
                var o = Object.create(ViewPrototype);
                extend(o, attrs);
                return o;
            }
        };
    }());

So, really, your imagination is the limit. Of course, you'll want the spec file to be the standard Jasmine spec code, which is what I have above; but you can tweak that however you like as well.

The next step is to add the script tags for these files to SpecRunner.html. At first, this might seem tricky: how can we add lines to the middle of a file programmatically? Once again, it's sed that does the job.

    sed -i "" "11a\
    <script src='src/$2.js'></script>\
    <script src='spec/$2Spec.js'></script>
    " SpecRunner.html

We start just like we did before: in-place edit without a backup. Then our command: at line 11, we want to append the two following lines. It's important to escape the two new-lines, so that they'll appear in the text. As you can see, this just inserts those two scripts tags, exactly what we need for this step.

We can end with some output:

    echo "Created:"
    echo "	- src/$2.js"
    echo "	- spec/$2Spec.js"
    echo "Edited:"
    echo "	- SpecRunner.html"

And that's jazz create!


Step 7 – Running the Specs with jazz run

The last step is to actually run the tests. This means opening the SpecRunner.html file in a browser. There's a bit of a caveat here. On Mac OS X, we can use the open command to open a file in it's default program; this won't work on any other OS, but that's the way I do it here. Unfortunately, there's no real cross-platform way to do this, that I know of. If you're using this script under Cygwin on Windows, you can use cygstart in place of open; otherwise, try googling "[your OS] shell script open browser" and see what you come up with. Unfortunately, some versions of Linux (at least Ubuntu, in my experience) have an open command that's for something completely different. All this to say, your mileage with the following may vary.

if [ "`which open`" = '/usr/bin/open' ] 
then
    open SpecRunner.html
else
    echo "Please open SpecRunner.html in your browser"
fi

By now, you know exactly what this does: if we have open, we'll open SpecRunner.html, otherwise, we'll just print a message telling the user to open the file in the browser.

Originally, that if condition looked like this:

if which open > /dev/null

As we did with xattr, it just checked for existence of open; however, since I found out that there's a different open command on Linux (even on my Ubuntu server, which can't even open a browser!), I figured it might be better to compare the path of the open program, since the Linux one is at /bin/open (again, at least on Ubuntu server).

All this extra wordiness about open might sound like an excuse for my lack of a good solution, it actually points out something important about the command line. Don't mistake understanding the terminal with understanding a computer's configuration. This tutorial, and the associated course, have taught you a little bit more about the Bash shell (and the Z shell), but that doesn't mean each computer you use will be configured the same; there are many ways to install new commands (or different versions of commands), as well as remove commands. Caveat developer.

Well, that's the entire script! Here it is again, all together:

    #! /bin/sh

    function help () {
        echo "jazz - A simple script that makes using the Jasmine testing framework in a standalone project a little simpler."
        echo ""
        echo "    jazz init                  - include jasmine in the project";
        echo "    jazz create FunctionName   - creates ./src/FunctionName.js ./spec/FunctionNameSpec.js";
        echo "    jazz run                   - runs tests in browser";
    }

    JASMIME_LINK="http://cloud.github.com/downloads/pivotal/jasmine/jasmine-standalone-1.3.1.zip"

    if [ $1 ] 
    then
      case "$1" in
      init)
        echo "Downloading Jasmine . . ."
        curl -sO $JASMIME_LINK 
        unzip -q `basename $JASMIME_LINK` 
        rm `basename $JASMIME_LINK` src/*.js spec/*.js
        
        if which xattr > /dev/null && [ "`xattr SpecRunner.html`" = "com.apple.quarantine" ]
        then
          xattr -d com.apple.quarantine SpecRunner.html
        fi

        sed -i "" "12,18d" SpecRunner.html
        echo "Jasmine initialized!"
      ;;
      create)
        if [ $2 ]
        then 
          echo "function $2 () {

}" > ./src/$2.js
          echo "describe('$2', function () {
it('runs');
});" > ./spec/$2Spec.js
          sed -i "" "11a\
          <script src='src/$2.js'></script>\
          <script src='spec/$2Spec.js'></script>
          " SpecRunner.html
          echo "Created:"
          echo "	- src/$2.js"
          echo "	- spec/$2Spec.js"
          echo "Edited:"
          echo "	- SpecRunner.html"
        else
          echo 'please add a name for the file'
        fi
      ;;
      "run")
        if [ "`which open`" = '/usr/bin/open' ] 
        then
          open ./SpecRunner.html
        else
          echo "Please open SpecRunner.html in your browser"
        fi
      ;;
      *)
        help;
      ;;
      esac
    else
      help;
    fi

Well, go on, give it a try!

    mkdir project
    cd project
    jazz init
    jazz create Dog
    # edit src/Dog.js and spec/DogSpec.js
    jazz run

By the way, if you want to have some more fun with this project, you can find it on Github.


Conclusion

So there you have it! We've just written an intermediate level shell script; that wasn't so bad, now, was it? Don’t forget to stay tuned for my upcoming Tuts+ Premium course; you'll learn a lot more about many of the techniques used in this article, as well as countless others. Have fun on the terminal!

April 16 2013

18:41

IRC is Back: Here’s Your Starter Guide

How of many of you remember Compuserve? What about Prodigy? I'm asking because, back in the good ‘ole days, they were popular services that allowed you to communicate with other people from all over the world. In many cases, they offered complementary services, such as forums or classified services, not to mention a pretty appealing user interface (for the time).

Some of you may be too young to even know about those services, but I'll go out on a limb and assume that everyone's used either AOL Instant Messenger, Skype, or Yahoo! Messenger at some point. The main thing is that they were meant to facilitate communication. In the ever-growing world of remote teams and worldwide collaboration on projects, the ability to get good, immediate feedback is critical to solving problems. Interestingly enough, all of the services that I noted above, while awesome in their own way, have taken a backseat to another medium for developers: IRC.


Everything Old is New Again

What I've noticed in the last couple of years is a resurgence in using IRC as a primary means of communication.

IRC (Internet Relay Chat) is a protocol, created in 1988, and was meant to help facilitate group discussions, via various channels. The channels served to identify different discussion topics you could pop in on. Having been on the Internet for some time now (I used to dial-in with a 2400 baud modem), I remember using IRC, but gravitated to many of the services that I mentioned earlier, because they just seemed more user-friendly and feature-rich.

What I've noticed in the last couple of years is a resurgence in using IRC as a primary means of communication, especially for open source projects. While there have been group-based services for a while no (Skype, Campfire, Google Hangouts, etc.), the relatively open nature of IRC makes it easy for large groups of people to hop on a channel and collaborate easily.

I'm also going to say (and I'm sure I'll get some grief for it) that I think there's a little bit of "geek cred" action going on here when you can tell someone, "let's chat about it on IRC".

So you want to get into this "IRC thing". That's actually a good thing because to be honest, a LOT of great discussions are happening on there and in some cases, going to a channel is the only way to get any decent support, especially for some open source efforts. So let's first break down how things work.


Picking the Right Network

The first thing that you need to understand is the lay of the land. IRC works by using some type of client that connects you to an IRC network. The networks are just machines that are setup to allow users to connect to IRC. There are a number of networks out there, and most are targeted to specific interests. For example, Aniverse is geared toward the Anime crowd, while QuakeNet targets video game players – especially Quakeworld players. According to IRCHelp.org, the major networks are as follows:

  • EFNet

  • Undernet

  • Dalnet

  • Quakenet

  • Freenode

These are networks that average 10,000 or more users, and tend to have the largest, busiest channels. There are also regional networks for different countries as well, but in terms of development, from my experience, most developers tend to jump on Freenode – and rightfully so. Here's how the Freenode network describes itself:

"An IRC network providing discussion facilities for the Free and Open Source Software communities, not-for-profit organizations, and related communities."

And with open source software so vitally important today to just about every major web property, it makes perfect sense why so many of the popular development channels are on Freenode. Wikipedia lists it as the largest of the major networks, with approximately 85k users on it at peak hours. It's the network I'll be focusing on in this article.


Tune in to the Best Channels

IRC does support 1 to 1 private chats.

I referenced the termm "channels" before; just as when you're watching TV, specific networks have specific channels for specific topics. By joining a channel, you're choosing to jump into a discussion group about a specific topic. The discussion is typically free-form so don't expect to go into a threaded UX with everything tightly organized or hierarchical. In fact, depending on the number of users in the channel and how active they are, discussions can be challenging to piece together. That description isn't meant to dissuade you from using IRC, but more to get you mentally prepared, especially if you're used to highly organized mediums like online forums or threaded email discussion lists. IRC is a wide-open, free-for-all discussion medium so you'll certainly see a number of discussions being carried on at once, in some cases overlapping. The key thing is to have patience so that you can filter through the noise and get your interests met.

You'll typically participate in group chats, but IRC does support 1 to 1 private chats as well, along with the ability to transfer files. As always, safe Internet practices are essential. Chances are high that you DON'T know the person on the other end of the wire; so you need to be careful when accepting anything from anyone. In addition, IRC allows scripts to be run, which provide additional capabilities. Again, be safe and DO NOT run any scripts from anyone unless you're absolutely 100% sure of what it's meant to do. As a general rule, I don't accept any files or run any scripts through IRC. I don't think IRC is any more insecure than other services on the Internet, but I'm hyper-vigilant because I probably won't know the person sending me something. Common sense in my opinion.

I recently asked publicly which channels developers are using and I got some nice feedback. The following list is a little long, but I think worth posting since the channels are incredibly useful:

  • #html5
  • #whatwg
  • #gruntjs
  • #yeoman
  • ##javascript
  • #jquery
  • #angularjs
  • #requirejs
  • #node.js
  • #css
  • #httpd
  • #svg
  • #webgl
  • #webrtc

This is NOT an all-encompassing list of every awesome web development channel so if you think there are others that would be useful, drop them in the comments section.

The hash ("#") in front of the channel is purposeful and meant to identify channels that are available across a whole network.

One thing to note is that channels are typically managed by channel ops (short for operators). These people help to set the channel up, and generally keep things running smooth. They also have the power to manage users on the channel and can kick or ban you if you get out of line. You don't need to pay them homage or anything, but do understand that, in spite of it's openness, IRC is a managed service and most channels, especially those for open source efforts, will expect a certain level of courtesy and professionalism so be cool.


Getting Connected

The quickest way to get onto IRC is installing client software. Whether you use Windows, OSX or Linux, there are a number of great choices available to you. I've used clients on both Windows and OSX and find the following to be great options:

Windows

HexChat – This is my client of choice for Windows. It's a fork of the popular XChat client, but is actively maintained and updated. It's also open source and freely available. I was using XChat before and even purchased a license for it, but I haven't seen any development activity on it since Aug, 2010 and can only assume that it's not being supported any longer. HexChat has stepped up and taken XChat's place, at least for me.

alt text

mIRC – mIRC has been around forever and is a solid IRC client. I used it many years ago, and it's probably the most widely used Windows-based client available. It does require user registration after 30-days if I recall – and that runs $20 – but it's a one-time, lifetime registration fee that, according to the site, "will work with current and future versions of mIRC." I'm certainly not averse to spending money on good software, so I may revisit mIRC in the future.

OSX

Colloquy – This is my client of choice for OSX. I find the user interface to be much more intuitive than other software, like LimeChat. It could be that it reminds me of HexChat, which makes switching between OSes easier. Either way, Colloquy is the best OSX-based IRC client. It's also open source and there's an app available for the iPhone, which is very cool.

LimeChat – Seems like all the cool kids are using this now. I gave it a run and it's not bad; however, I don't think it's better than Colloquy. There was something about the way the UI was structured that "just didn't feel right" to me. Like Colloquy, it's open source and also has an app available for iOS.

Textual is also a new alternative that seems to be quite good.

Browser-based

Don't want to install software? No problem. Just hop on over to Freenode's Web Chat. Using any browser, you should be able to jump onto Freenode and visit channels of your choice. The only downside is that you can't store preferences so if you want to visit a set of channels regularly, it can become cumbersome re-entering your channels.

Here's the login screen:

alt text

And here's the channel's chat interface:

alt text

If you look at the this screenshot and compare it to that of HexChat or Colloquy, you'll see that, from a usability perspective, using a dedicated client is MUCH easier, since it lets you organize and connect to your channels every time you startup the program. If we look at HexChat, for example, I can easily choose the network to connect to:

alt text

as well as define the default channels I'd like to join everytime I connect to Freenode:

alt text

While the web interface for Freenode makes it convenient to connect from any device without installing anything, it does limit me specifically to Freenode, and forces me to manually join each channel I want to participate in using the "/join" command like this:

/join #jquery

If you have a list of IRC channels as long as Paul Irish's, typing all of this in over and over will get boring really quick:

alt text

In HexChat, all I have to do is specify a comma-delimited list of my channels in the "Favorite channels" field for the network and when I connect, I get auto-joined:

#bocoup,#cordova,#css,#emberjs,#grunt,#html5,#inimino,##javascript,#jquery,#node.js,#requirejs,#rubyonrails,#webgl,#whatwg,#yeoman

Much easier.


Getting Your Own Nick

When you join a network, you're going to have to identify yourself using a nickname, commonly referred to as a “nick.” It's really no different than creating a username on your favorite site, and helps people to get to know you as you participate in discussions. I recommend not only getting a nick, but password protecting it so that someone else won't take it from you (accidentally or purposefully) later on.

Setting up a nick is really simple. First, you specify the nickname that you'd like to use, and then connect to Freenode. Once you've connected to the network, you'll use the following command to create your password and associate your email address to your nickname:

/msg NickServ REGISTER password youremail@example.com

You would substitute "password" with your password and "youremail@example.com" with the email you'd like to associate to your nick. You'll then get the following message in your IRC client (I've blocked out my password for obvious reasons):

[14:16] NickServ REGISTER ***** reybango@yahoo.com [14:16] -NickServ- An email containing nickname activation instructions has been sent to reybango@yahoo.com. [14:16] -NickServ- If you do not complete registration within one day, your nickname will expire. [14:16] -NickServ- bangoboom is now registered to reybango@yahoo.com, with the password *****.

You're not done, as Freenode will send you an email with an activation command. Here's what I got:

/msg NickServ VERIFY REGISTER bangoboom cikdmbzhrflr

I put that into the Freenode IRC command line, which returned the following message:

[14:20] NickServ VERIFY REGISTER bangoboom cikdmbzhrflr [14:20] -NickServ- bangoboom has now been verified. [14:20] -NickServ- Thank you for verifying your e-mail address! You have taken steps in ensuring that your registrations are not exploited.

The last sentence reinforces what I noted earlier:

"You have taken steps in ensuring that your registrations are not exploited."

Your nick is your identity on IRC, and you should protect it as best as possible.


Common Commands You Should Know

Once you're in the mix and chatting away, there are some commands that will make your life easier. I'll list them below:

  • /who <nick> – This allows you to get more information about someone in a channel.

  • /list – This will return a list of all channels available on a network.

  • /join – This allows you to join a channel via the Freenode command line. Remember to prefix the channel name with a hash ("#")

  • /msg – This allows you to have a private chat with the person whose nick you've specified and will send them the message to kick off the chat

  • /invite – This allows you to invite a user to another channel for a chat.

  • /away – Tells users that you're away from your PC for a bit

  • /quit – Tells the network you're done and are leaving


Etiquette (or How Not to be a Jerk)

It should be common sense by now, but, unfortunately, the perception of anonymity on the Internet leads people to believe that they can act any way they like. Just remember that, no matter what, you're talking to another person on the other end of that wire. So my typical advice is don't act online like you wouldn't act in person. Don’t forget that most channels have channel ops, and they will boot you if you get out of hand. Don't be "that guy".

In terms of how to communicate, here are some of the things I've observed:

  • Don't use all caps. Should be obvious but you'd be surprised.

  • If you have a question, try to be as detailed as possible. Trying to decipher an issue online is tough.

  • Creating a reduced test case before posting in a channel will get you much farther in solving your problem quickly. The community will also appreciate the fact that you've taken the time to narrow down the issue.

  • Avoid flooding a channel with a ton of noise. What I mean is do your best to keep the discussion brief and on-topic so that it benefits everyone.

Some channels will have banner messages when you first come into it. READ THEM as they're generally there to outline channel guidelines or offer important updates about the channel or project. Here's the banner from the jQuery channel:

"jQuery 1.9.1 http://jquery.com | jQuery UI 1.10.2 http://jqueryui.com | jQuery Mobile 1.3.0 http://jquerymobile.com | Docs: http://api.jquery.com http://learn.jquery.com | Paste code at http://jsfiddle.net or http://jsbin.com | Dev Channels #jquery-dev #jqueryui-dev #jquerymobile-dev | http://try.jquery.com for learning the basics | Channel publicly logged to http://irc.jquery.org/"

Notice that it tells you the most current versions of the libs, and offers links to resources to help solve your jQuery-related issues.


Being Safe

I can't stress enough that IRC offers no anonymity. Don't assume anything you say is private and can't be seen (unless you've gone to great lengths to anonymize yourself). Conversations can be logged and in fact, most IRC clients have that feature built-in. Your IP address is also easily visible by simply using the "/who" command.

As I mentioned above:

  • Don't accept file transfers

  • Don't accept direct connection requests (you're bypassing the IRC server and directly connecting to another computer)

  • Don't run commands that someone tells you to run

  • Don't run scripts that someone has sent you. You can get backdoored.

This is my own personal feelings and you can do as you please. If you think someone is being malicious, report it to a channel op so they can check it out.

This page offers a great outline of what to look out for so you don't get burned.


Getting Involved

IRC is the new black, when it comes to online communication and it's important that, as times change, we adjust to how people are talking with one another. IRC offers a great opportunity to get developers from across the world together and share a wealth of knowledge. The interface is simple and explicit, making it incredibly easy to pick up and be productive. And with the numerous clients available, getting involved is straightforward. So, if you haven't yet tried IRC or you're a veteran that needs to re-grease the wheels, IRC is back and ready for you.

April 11 2013

14:23

PSR-Duh!

In a previous lesson here on Nettuts+, you learn about PSR; however, that article didn’t detail the process of integrating that coding style into your projects. Let’s fix that!

Note: this article assumes that you’ve read PSR-Huh?, and understand what PSR refers to. Let’s begin with the first standard: PSR-0.


PSR-0 – The Autoloading Standard

The PHPCS plugin is the most helpful tool I’ve used.

In the past, we included PHP files in one of two ways:

  • Using a giant block of include statements at the top of each file.
  • List all includes in a single file and include that single file within your project.

There are pros and cons to both of these approaches, but, I think we can all agree that neither are optimal or modern solutions. PHP5 introduced the concept of autoloading files based on their class names; so, PSR-0 aims to keep filenames consistent.

Namespaces have nothing to do with filenames or autoloading; you can technically declare different namespaces in the same file. For example, the following code is perfectly valid.

<?php
namespace Nettuts;

Class Hello
{
    public function __construct()
    {
        echo "Nettuts+";
    }
}

namespace Gabriel;

Class Hello
{
    public function __construct()
    {
        echo "Gabriel";
    }
}

$h = new NettutsHello();
$h = new GabrielHello();

There are two Hello classes in this single file, but they reside within different namespaces. The final two lines of this code instantiate the Hello() classes on their respective namespaces. The first outputs “Nettuts+”, while the second echos “Gabriel.” Namespaces allow you to differentiate between two classes with the same name, much like you might be used to with folders on your desktop. The PSR-0 standard simply leverages the benefits of namespaces, making it easy to autoload your classes. By consistently naming your files, you can create a function that locates the necessary files automatically.

To be PSR-1 compliant, you also must follow PSR-0.

Be sure to read the full standard, but to summarize:

  • Each class must be namespaced with the project’s (or creator’s) name.
  • Underscores within the class’ name should be converted to directory separators.
  • Files must have the .php extension.

For example, a class reference of:

NettutsDatabaseSQL_Postgres

if following PSR-0, should translate to this path:

./Nettuts/Database/SQL/Postgres.php

How might we implement this functionality? The most obvious solution is to use Composer, which ships with a PSR-0 compliant autoloader. If you leverage Composer in your projects (and you should), then opt for its autoloader, rather than writing your own.

A PSR-0 compliant loader allows you to specify a base path, informing the loader which directory to look in first. To get started, create a simple composer.json file that contains the following JSON:

{
    "autoload": {
        "psr-0": {
            "Nettuts": "./",
            "Gmanricks": "vendor/"
        }
    }
}

This JSON file tells Composer that we want to use the PSR-0 standard to autoload all Nettuts-namespaced files with the current directory (the root folder) as the base path. We also want to autoload all classes with the Gmanricks namespace, relative to the vendor folder (e.g. ./vendor/Gmanricks/ClassName).

Now, type “composer install” to generate the autoload classes, or “composer dump-autoload” on subsequent edits to regenerate the autoload classes. Also, don’t forget to require the autoloader somewhere early in your project.

<?php

require 'vendor/autoload.php';

Composer is your best option, but there may be scenarios when you want a small, simple autoloader. The PHP-FIG provides a sample autoloader that you can use:

function __autoload($className)
{
    $className = ltrim($className, '\');
    $fileName  = '';
    $namespace = '';
    if ($lastNsPos = strrpos($className, '\')) {
        $namespace = substr($className, 0, $lastNsPos);
        $className = substr($className, $lastNsPos + 1);
        $fileName  = str_replace('\', DIRECTORY_SEPARATOR, $namespace) . DIRECTORY_SEPARATOR;
    }
    $fileName .= str_replace('_', DIRECTORY_SEPARATOR, $className) . '.php';

    require $fileName;
}

It’s important to note that this loader attempts to load all classes using the PSR standard in the current directory.

Now that we’re successfully autoloading classes, let’s move on to the next standard: the basic coding standard.


PSR-1 – The Basic Coding Standard

PSR-1 defines general coding guidelines, which can be divided into two parts.

Naming Conventions

Namespaces allow you to differentiate between two classes with the same name.

As with any programming language, following naming conventions ultimately makes your code easier to read and maintain. Here’s a few rules to follow:

  • Class names use PascalCase.
  • Method names should be in camelCase.
  • Constants require all capital letters, separating each word with an underscore (e.g. CONSTANT_VARIABLE).

Code Conventions:

There’s more to it than naming conventions; follow these guidelines as well:

  • Only use <?php or <?= in your code. Don’t close PHP within a class.
  • Files should either declare symbols or use them.
  • Files must be in UTF-8 format without BOM for PHP code

Most of these are self-explanatory, but the middle convention is slightly confusing. It essentially dictates that any declaration, be it functions, classes, etc., should be separated into their own files. This not only promotes best practices like code-reuse and separation, but it keeps your code neat and clean.

It’s worth mentioning that each PSR standard builds upon the previous PSR standard. As such, to be PSR-1 compliant, you also must follow PSR-0. By following these two standards, your code will be properly namespaced and autoloaded. There really isn’t a reason not to follow them.

Yes, some developers complain about PSR and prefer to follow other conventions, but by following this standard, you can share code with everyone without worrying about its consistency. Having said that, nobody is forcing your hand here. It’s simply a recommended guideline.

The next standard, PSR-2, dives into the specifics of how you should structure your code.


PSR-2 – The Advanced Coding Standard

PSR-2 dives into the specifics of how you should structure your code.

Next, we come to the one standard that PHP developers struggle with most; in fact, it’s the reason why I chose to write this article.

PSR-2 defines many rules, many of which are listed below:

  • Four spaces should be used instead of tabs.
  • The ideal line length should be under 80 characters, but a soft limit of 120 characters should be imposed on all lines.
  • There should be one blank line under the namespace and use declarations.
  • A method’s or class’ opening brace must be on its own line.
  • A method’s or class’ closing brace must go on the line immediately after the body.
  • All properties and methods require a visibility level.
  • The ‘abstract‘ / ‘final‘ keywords should appear before the visibility while ‘static‘ goes after.
  • Control structure keywords must be followed by one space.
  • A control statement’s opening brace should appear on the same line as the statement.

Be sure to view the entire spec for a complete overview.

PSR-2 is just as important as PSR-1 (and PSR-0). It intends to make code easy to read and maintain. But, as they say, “The devil is in the details.” There are a lot of details to remember, which can be difficult if your programming habits differ from what the standard defines. Thankfully, if you’re on board, there are tools that help you adhere to PSR-0, PSR-1 and PSR-2. Perhaps the best tool is the Sublime Text plugin, PHPCS.


PHPCS – PHP Code Sniffer

The PHPCS plugin is the most helpful tool I’ve used, when it comes to getting code into shape. It allows you to not only ensure that your code follows the PSR standards, but it also uses PHP’s linter to check for syntax errors. This is a great time saver; you no longer have to worry about syntax errors when you test your code in the browser.

Install the package through Sublime Package Control (it’s called Phpcs), or, alternatively, with Git, using the following commands:

cd ~/Library/Application Support/Sublime Text 2/Packages/
git clone git://github.com/benmatselby/sublime-phpcs.git Phpcs

This installs the plugin, but you need a few dependencies before you can configure PHPCS. Once again, the easiest way to install them is with Composer. Browse to a directory of your choice and create a composer.json file with the following JSON:

{
    "name": "Nettuts PHPCS Demo",
    "require": {
        "squizlabs/php_codesniffer": "*",
        "fabpot/php-cs-fixer": "*",
        "phpmd/phpmd": "*"
    }
}

This installs the three dependencies into the current folder. Open a terminal window to your installation location and type composer install, and it will download the necessary packages.

Now you can configure the plugin in Sublime Text. Navigate to ‘Preferences’ > ‘Package Settings’ > ‘PHP Code Sniffer’ > ‘Settings – User’.

PHP Code-sniffer Settings

The plugin needs to know where the three dependencies reside, as well as the standard that we want our code to adhere to:

{
    "phpcs_additional_args": {
        "--standard": "PSR2",
        "-n": ""
    },
    "phpcs_executable_path": "DEPENDENCY_PATH/vendor/bin/phpcs",
    "phpmd_executable_path": "DEPENDENCY_PATH/vendor/bin/phpmd",
    "php_cs_fixer_executable_path": "DEPENDENCY_PATH/vendor/bin/php-cs-fixer"
}

These settings inform PHPCS that we want to ahere to the PSR2 standard and provide each dependency’s path. Don’t forget to replace DEPENDENCY_PATH with your actual path.

Restart Sublime, and the code sniffer will scan your code when you save your PHP files.

Phpcs Sublime Extension Errors

Right-clicking in the editor will also list several new options, such as clearing error marks and attempting to fix the non-standard issues. However, considering that the point of this article is to get you used to the standard, I suggest manually fixing your code and avoiding the automatic fixer feature.


Conclusion

The PSR standards were created so that code could easily be reused from project to project, without sacrificing on code style consistency. While they may feel overwhelming at first, you can use the ideas and tools from this article to help you make the transition.

To reiterate one last time: nobody is forcing you to change the way that you code in PHP. It’s simply a guide, originally meant for framework interoperability. That said, at Nettuts+, we consider it a best practice to follow. Now make up your own mind! If you have any questions, let’s hear them below!

April 08 2013

13:00

HTTP: The Protocol Every Web Developer Must Know – Part 1

HTTP stands for Hypertext Transfer Protocol. It’s a stateless, application-layer protocol for communicating between distributed systems, and is the foundation of the modern web. As a web developer, we all must have a strong understanding of this protocol.

Let’s review this powerful protocol through the lens of a web developer. We’ll tackle the topic in two parts. In this first entry, we’ll cover the basics and outline the various request and response headers. In the follow-up article, we’ll review specific pieces of HTTP – namely caching, connection handling and authentication.

Although I’ll mention some details related to headers, it’s best to instead consult the RFC (RFC 2616) for in-depth coverage. I will be pointing to specific parts of the RFC throughout the article.


HTTP Basics

HTTP allows for communication between a variety of hosts and clients, and supports a mixture of network configurations.

To make this possible, it assumes very little about a particular system, and does not keep state between different message exchanges.

This makes HTTP a stateless protocol. The communication usually takes place over TCP/IP, but any reliable transport can be used. The default port for TCP/IP is 80, but other ports can also be used.

Request/Response Pair

Custom headers can also be created and sent by the client.

Communication between a host and a client occurs, via a request/response pair. The client initiates an HTTP request message, which is serviced through a HTTP response message in return. We will look at this fundamental message-pair in the next section.

The current version of the protocol is HTTP/1.1, which adds a few extra features to the previous 1.0 version. The most important of these, in my opinion, includes persistent connections, chunked transfer-coding and fine-grained caching headers. We’ll briefly touch upon these features in this article; in-depth coverage will be provided in part two.

URLs

At the heart of web communications is the request message, which are sent via Uniform Resource Locators (URLs). I’m sure you are already familiar with URLs, but for completeness sake, I’ll include it here. URLs have a simple structure that consists of the following components:

URL Structure

The protocol is typically http, but it can also be https for secure communications. The default port is 80, but one can be set explicitly, as illustrated in the above image. The resource path is the local path to the resource on the server.

Verbs

There are also web debugging proxies, like Fiddler on Windows and Charles Proxy for OSX.

URLs reveal the identity of the particular host with which we want to communicate, but the action that should be performed on the host is specified via HTTP verbs. Of course, there are several actions that a client would like the host to perform. HTTP has formalized on a few that capture the essentials that are universally applicable for all kinds of applications.

These request verbs are:

  • GET: fetch an existing resource. The URL contains all the necessary information the server needs to locate and return the resource.
  • POST: create a new resource. POST requests usually carry a payload that specifies the data for the new resource.
  • PUT: update an existing resource. The payload may contain the updated data for the resource.
  • DELETE: delete an existing resource.

The above four verbs are the most popular, and most tools and frameworks explicitly expose these request verbs. PUT and DELETE are sometimes considered specialized versions of the POST verb, and they may be packaged as POST requests with the payload containing the exact action: create, update or delete.

There are some lesser used verbs that HTTP also supports:

  • HEAD: this is similar to GET, but without the message body. It’s used to retrieve the server headers for a particular resource, generally to check if the resource has changed, via timestamps.
  • TRACE: used to retrieve the hops that a request takes to round trip from the server. Each intermediate proxy or gateway would inject its IP or DNS name into the Via header field. This can be used for diagnostic purposes.
  • OPTIONS: used to retrieve the server capabilities. On the client-side, it can be used to modify the request based on what the server can support.

Status Codes

With URLs and verbs, the client can initiate requests to the server. In return, the server responds with status codes and message payloads. The status code is important and tells the client how to interpret the server response. The HTTP spec defines certain number ranges for specific types of responses:

1xx: Informational Messages

All HTTP/1.1 clients are required to accept the Transfer-Encoding header.

This class of codes was introduced in HTTP/1.1 and is purely provisional. The server can send a Expect: 100-continue message, telling the client to continue sending the remainder of the request, or ignore if it has already sent it. HTTP/1.0 clients are supposed to ignore this header.

2xx: Successful

This tells the client that the request was successfully processed. The most common code is 200 OK. For a GET request, the server sends the resource in the message body. There are other less frequently used codes:

  • 202 Accepted: the request was accepted but may not include the resource in the response. This is useful for async processing on the server side. The server may choose to send information for monitoring.
  • 204 No Content: there is no message body in the response.
  • 205 Reset Content: indicates to the client to reset its document view.
  • 206 Partial Content: indicates that the response only contains partial content. Additional headers indicate the exact range and content expiration information.

3xx: Redirection

404 indicates that the resource is invalid and does not exist on the server.

This requires the client to take additional action. The most common use-case is to jump to a different URL in order to fetch the resource.

  • 301 Moved Permanently: the resource is now located at a new URL.
  • 303 See Other: the resource is temporarily located at a new URL. The Location response header contains the temporary URL.
  • 304 Not Modified: the server has determined that the resource has not changed and the client should use its cached copy. This relies on the fact that the client is sending ETag (Enttity Tag) information that is a hash of the content. The server compares this with its own computed ETag to check for modifications.

4xx: Client Error

These codes are used when the server thinks that the client is at fault, either by requesting an invalid resource or making a bad request. The most popular code in this class is 404 Not Found, which I think everyone will identify with. 404 indicates that the resource is invalid and does not exist on the server. The other codes in this class include:

  • 400 Bad Request: the request was malformed.
  • 401 Unauthorized: request requires authentication. The client can repeat the request with the Authorization header. If the client already included the Authorization header, then the credentials were wrong.
  • 403 Forbidden: server has denied access to the resource.
  • 405 Method Not Allowed: invalid HTTP verb used in the request line, or the server does not support that verb.
  • 409 Conflict: the server could not complete the request because the client is trying to modify a resource that is newer than the client’s timestamp. Conflicts arise mostly for PUT requests during collaborative edits on a resource.

5xx: Server Error

This class of codes are used to indicate a server failure while processing the request. The most commonly used error code is 500 Internal Server Error. The others in this class are:

  • 501 Not Implemented: the server does not yet support the requested functionality.
  • 503 Service Unavailable: this could happen if an internal system on the server has failed or the server is overloaded. Typically, the server won’t even respond and the request will timeout.

Request and Response Message Formats

So far, we’ve seen that URLs, verbs and status codes make up the fundamental pieces of an HTTP request/response pair.

Request-Response details

Let’s now look at the content of these messages. The HTTP specification states that a request or response message has the following generic structure:

message = <start-line>
          *(<message-header>)
          CRLF
          [<message-body>]

<start-line> = Request-Line | Status-Line 
<message-header> = Field-Name ':' Field-Value

It’s mandatory to place a new line between the message headers and body. The message can contain one or more headers, of which are broadly classified into:

The message body may contain the complete entity data, or it may be piecemeal if the chunked encoding (Transfer-Encoding: chunked) is used. All HTTP/1.1 clients are required to accept the Transfer-Encoding header.

General Headers

There are a few headers (general headers) that are shared by both request and response messages:

general-header = Cache-Control            
               | Connection        
               | Date              
               | Pragma            
               | Trailer           
               | Transfer-Encoding 
               | Upgrade           
               | Via               
               | Warning

We have already seen some of these headers, specifically Via and Transfer-Encoding. We will cover Cache-Control and Connection in part two.

The status code is important and tells the client how to interpret the server response.

  • Via header is used in a TRACE message and updated by all intermittent proxies and gateways
  • Pragma is considered a custom header and may be used to include implementation-specific headers. The most commonly used pragma-directive is Pragma: no-cache, which really is Cache-Control: no-cache under HTTP/1.1. This will be covered in Part 2 of the article.
  • The Date header field is used to timestamp the request/response message
  • Upgrade is used to switch protocols and allow a smooth transition to a newer protocol.
  • Transfer-Encoding is generally used to break the response into smaller parts with the Transfer-Encoding: chunked value. This is a new header in HTTP/1.1 and allows for streaming of response to the client instead of one big payload.

Entity headers

Request and Response messages may also include entity headers to provide meta-information about the the content (aka Message Body or Entity). These headers include:

entity-header  = Allow                    
               | Content-Encoding  
               | Content-Language  
               | Content-Length    
               | Content-Location  
               | Content-MD5       
               | Content-Range     
               | Content-Type      
               | Expires           
               | Last-Modified  

All of the Content- prefixed headers provide information about the structure, encoding and size of the message body. Some of these headers need to be present if the entity is part of the message.

The Expires header indicates a timestamp of whent he entity expires. Interestingly, a “never expires” entity is sent with a timestamp of one year into the future. The Last-Modified header indicates the last modification timestamp for the entity.

Custom headers can also be created and sent by the client; they will be treated as entity headers by the HTTP protocol.

This is really an extension mechanism, and some client-server implementations may choose to communicate specifically over these extension headers. Although HTTP supports custom headers, what it really looks for are the request and response headers, which we cover next.

Request Format

The request message has the same generic structure as above, except for the request line which looks like:

Request-Line = Method SP URI SP HTTP-Version CRLF
Method = "OPTIONS"
       | "HEAD"  
       | "GET"  
       | "POST"  
       | "PUT"  
       | "DELETE"  
       | "TRACE"

SP is the space separator between the tokens. HTTP-Version is specified as “HTTP/1.1″ and then followed by a new line. Thus, a typical request message might look like:

GET /articles/http-basics HTTP/1.1
Host: www.articles.com
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Note the request line followed by many request headers. The Host header is mandatory for HTTP/1.1 clients. GET requests do not have a message body, but POST requests can contain the post data in the body.

The request headers act as modifiers of the request message. The complete list of known request headers is not too long, and is provided below. Unknown headers are treated as entity-header fields.

request-header = Accept                   
               | Accept-Charset    
               | Accept-Encoding   
               | Accept-Language   
               | Authorization     
               | Expect            
               | From              
               | Host              
               | If-Match          
               | If-Modified-Since 
               | If-None-Match     
               | If-Range          
               | If-Unmodified-Since
               | Max-Forwards       
               | Proxy-Authorization
               | Range              
               | Referer            
               | TE                 
               | User-Agent

The Accept prefixed headers indicate the acceptable media-types, languages and character sets on the client. From, Host, Referer and User-Agent identify details about the client that initiated the request. The If- prefixed headers are used to make a request more conditional, and the server returns the resource only if the condition matches. Otherwise, it returns a 304 Not Modified. The condition can be based on a timestamp or an ETag (a hash of the entity).

Response Format

The response format is similar to the request message, except for the status line and headers. The status line has the following structure:

Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF
  • HTTP-Version is sent as HTTP/1.1
  • The Status-Code is one of the many statuses discussed earlier.
  • The Reason-Phrase is a human-readable version of the status code.

A typical status line for a successful response might look like so:

HTTP/1.1 200 OK

The response headers are also fairly limited, and the full set is given below:

 response-header = Accept-Ranges
                 | Age
                 | ETag              
                 | Location          
                 | Proxy-Authenticate
                 | Retry-After       
                 | Server            
                 | Vary              
                 | WWW-Authenticate
  • Age is the time in seconds since the message was generated on the server.
  • ETag is the MD5 hash of the entity and used to check for modifications.
  • Location is used when sending a redirection and contains the new URL.
  • Server identifies the server generating the message.

It’s been a lot of theory upto this point, so I won’t blame you for drowsy eyes. In the next sections, we will get more practical and take a survey of the tools, frameworks and libraries.


Tools to View HTTP Traffic

There are a number of tools available to monitor HTTP communication. Here, we list some of the more popular tools.

Undoubtedly, the Chrome/Webkit inspector is a favorite amongst web developers:

Chrome Inspector

There are also web debugging proxies, like Fiddler on Windows and Charles Proxy for OSX. My colleague, Rey Bango wrote an excellent article on this topic.

Fiddler Charles Proxy

For the command line, we have utilities like curl, tcpdump and tshark for monitoring HTTP traffic.


Using HTTP in Web Frameworks and Libraries

Now that we have looked at the request/response messages, it’s time that we learn how libraries and frameworks expose it in the form of an API. We’ll use ExpressJS for Node, Ruby on Rails, and jQuery Ajax as our examples.

ExpressJS

If you are building web servers in NodeJS, chances are high that you’ve considered ExpressJS. ExpressJS was originally inspired by a Ruby Web framework, called Sinatra. As expected, the API is also equally influenced.

Because we are dealing with a server-side framework, there are two primary tasks when dealing with HTTP messages:

  • Read URL fragments and request headers.
  • Write response headers and body

Understanding HTTP is crucial for having a clean, simple and RESTful interface between two endpoints.

ExpressJS provides a simple API for doing just that. We won’t cover the details of the API. Instead, we will provide links to the detailed documentation on ExpressJS guides. The methods in the API are self-explanatory in most cases. A sampling of the request-related API is below:

On the way out to the client, ExpressJS provides the following response API:

  • res.status: set an explicit status code.
  • res.set: set a specific response header.
  • res.send: send HTML, JSON or an octet-stream.
  • res.sendFile: transfer a file to the client.
  • res.render: render an express view template.
  • res.redirect: redirect to a different route. Express automatically adds the default redirection code of 302.

Ruby on Rails

The request and response messages are mostly the same, except for the first line and message headers.

In Rails, the ActionController and ActionDispatch modules provide the API for handling request and response messages.

ActionController provides a high level API to read the request URL, render output and redirect to a different end-point. An end-point (aka route) is handled as an action method. Most of the necessary context information inside an action-method is provided via the request, response and params objects.

  • params: gives access to the URL parameters and POST data.
  • request: contains information about the client, headers and URL.
  • response: used to set headers and status codes.
  • render: render views by expanding templates.
  • redirect_to: redirect to a different action-method or URL.

ActionDispatch provides fine-grained access to the request/response messages, via the ActionDispatch::Request and ActionDispatch::Response classes. It exposes a set of query methods to check the type of request (get?(), post?(), head?(), local?()). Request headers can be directly accessed via the request.headers() method.

On the response side, it provides methods to set cookies(), location=() and status=(). If you feel adventurous, you can also set the body=() and bypass the Rails rendering system.

jQuery Ajax

Because jQuery is primarily a client-side library, its Ajax API provides the opposite of a server-side framework. In other words, it allows you to read response messages and modify request messages. jQuery exposes a simple API via jQuery.ajax(settings):

By passing a settings object with the beforeSend callback, we can modify the request headers. The callback receives the jqXHR (jQuery XMLHttpRequest) object that exposes a method, called setRequestHeader() to set headers.

$.ajax({
    url: 'http://www.articles.com/latest',
    type: 'GET',
    beforeSend: function (jqXHR) {
      jqXHR.setRequestHeader('Accepts-Language', 'en-US,en');
    }
  });
  • The jqXHR object can also be used to read the response headers with the jqXHR.getResponseHeader().
  • If you want to take specific actions for various status codes, you can use the statusCode callback:
$.ajax({
  statusCode: {
    404: function() {
      alert("page not found");
    }
  }
});

Summary

So that sums up our quick tour of the HTTP protocol.

We reviewed URL structure, verbs and status codes: the three pillars of HTTP communication.

The request and response messages are mostly the same, except for the first line and message headers. Finally, we reviewed how you can modify the request and response headers in web frameworks and libraries.

Understanding HTTP is crucial for having a clean, simple, and RESTful interface between two endpoints. On a larger scale, it also helps when designing your network infrastructure and providing a great experience to your end users.

In part two, we’ll review connection handling, authentication and caching! See you then.


References

March 28 2013

16:56

Guard is Your Best Friend

A common argument against using preprocessors, like Sass, is that they introduce too much complexity, and are difficult to get started with. Especially today, though, this couldn’t be further from the truth. In this episode, I’ll demonstrate my file monitoring tool of choice, Guard. I’ll think you’ll love it!


Screencast


Show Notes

  • Installed Gems:
    $ gem install guard
    $ gem install rb-fsevent
    $ gem install guard-sass
    $ gem install guard-coffeescript
    $ gem install guard-livereload
    $ gem install guard-concat
    $ gem install guard-uglify
    

    Alternatively, you can use multiple parameters when installing Gems: gem install guard guard-sass.

  • LiveReload Browser Extensions

March 14 2013

21:23

Pro Workflow in Laravel and Sublime Text

Not too long ago, I built a handful of generators for Laravel, which ease the process of various tasks. Today, thanks to help from Gaurav Narula, we’re turning things up a notch with the release of a new Sublime Text plugin that leverages the power of Artisan and the generators from directly within your editor.


Installation

There’s only a few steps to begin taking advantage of both of these tools.

  1. Install the Generator Commands for your Laravel application.
  2. Clone the Sublime Text plugin into your packages directory. (Soon to be available through Package Control).

That’s it! You should now be good to go! Search for “Laravel” within the command palette, and you’ll find a variety of options.

Laravel Generators

Let me show you my basic workflow below:

Questions? Bugs? Feature requests? Let us know below!

March 13 2013

18:09

Relational Databases for Dummies

Web apps can be split into two major components: a front-end that displays and collects information, and a back-end for storing the information. In this article, I’ll demonstrate what a relational database is, and how to properly design your database to store your app’s information.

A database stores data in an organized way so that it can be searched and retrieved later. It should contain one or more tables. A table is much like a spreadsheet, in that it’s made up of rows and columns. All rows have the same columns, and each column contains the data itself. If it helps, think of your tables in the same way that you would a table in Excel.

Fig. 1

Data can be inserted, retrieved, updated, and deleted from a table. The word, created, is generally used instead of inserted, so, collectively, these four functions are affectionately abbreviated as CRUD.

A relational database is a type of database that organizes data into tables, and links them, based on defined relationships. These relationships enable you to retrieve and combine data from one or more tables with a single query.

But that was just a bunch of words. To truly understand a relational database, you need to make one yourself. Let’s get started by getting some real data with which we can work.


Step 1: Get Some Data

In the spirit of the Nettuts+ Twitter clone articles (PHP, Ruby on Rails, Django), let’s get some Twitter data. I searched Twitter for “#databases” and took the following sample of ten tweets:

Table 1

full_nameusernametextcreated_atfollowing_username“Boris Hadjur”“_DreamLead”“What do you think about #emailing #campaigns #traffic in #USA? Is it a good market nowadays? do you have #databases?”“Tue, 12 Feb 2013 08:43:09 +0000″“Scootmedia”, “MetiersInternet”“Gunnar Svalander”“GunnarSvalander”“Bill Gates Talks Databases, Free Software on Reddit http://t.co/ShX4hZlA #billgates #databases”“Tue, 12 Feb 2013 07:31:06 +0000″“klout”, “zillow”“GE Software”“GEsoftware”“RT @KirkDBorne: Readings in #Databases: excellent reading list, many categories: http://t.co/S6RBUNxq  via @rxin Fascinating.”“Tue, 12 Feb 2013 07:30:24 +0000″“DayJobDoc”, “byosko”“Adrian Burch”“adrianburch”“RT @tisakovich: @NimbusData at the @Barclays Big Data conference in San Francisco today, talking #virtualization, #databases, and #flash memory.”“Tue, 12 Feb 2013 06:58:22 +0000″“CindyCrawford”, “Arjantim”“Andy Ryder”“AndyRyder5″“http://t.co/D3KOJIvF article about Madden 2013 using AI to prodict the super bowl #databases #bus311″“Tue, 12 Feb 2013 05:29:41 +0000″“MichaelDell”, “Yahoo”“Andy Ryder”“AndyRyder5″“http://t.co/rBhBXjma an article about privacy settings and facebook #databases #bus311″“Tue, 12 Feb 2013 05:24:17 +0000″“MichaelDell”, “Yahoo”“Brett Englebert”“Brett_Englebert”“#BUS311 University of Minnesota’s NCFPD is creating #databases to prevent “food fraud.” http://t.co/0LsAbKqJ”“Tue, 12 Feb 2013 01:49:19 +0000″“RealSkipBayless”, “stephenasmith”Brett Englebert“Brett_Englebert”“#BUS311 companies might be protecting their production #databases, but what about their backup files? http://t.co/okJjV3Bm”“Tue, 12 Feb 2013 01:31:52 +0000″“RealSkipBayless”, “stephenasmith”“Nimbus Data Systems”“NimbusData”“@NimbusData CEO @tisakovich @BarclaysOnline Big Data conference in San Francisco today, talking #virtualization, #databases,& #flash memory”“Mon, 11 Feb 2013 23:15:05 +0000″“dellock6″, “rohitkilam”“SSWUG.ORG”“SSWUGorg”“Don’t forget to sign up for our FREE expo this Friday: #Databases, #BI, and #Sharepoint: What You Need to Know! http://t.co/Ijrqrz29″“Mon, 11 Feb 2013 22:15:37 +0000″“drsql”, “steam_games”

Here’s what each column name means:

MySQL is used at just about every Internet company you have heard of.

  • full_name: The user’s full name
  • username: The Twitter handle
  • text: The tweet itself
  • created_at: The timestamp of the tweet
  • following_username: A list of people this user follows, separated by commas. For briefness, I limited the list length to two

This is all real data; you can search Twitter and actually find these tweets.

This is good. The data is all in one place; so it’s easy to find, right? Not exactly. There are a couple problems with this table. First, there is repetitive data across columns. The “username” and “following_username” columns are repetitive, because both contain the same type of data — Twitter handles. There is another form of repetition within the “following_username” column. Fields should only contain one value, but each of the “following_username” fields contain two.

Second, there is repetitive data across rows.

@AndyRyder5 and @Brett_Englebert each tweeted twice, so the rest of their information has been duplicated.

Duplicates are problematic because it makes the CRUD operations more challenging. For example, it would take longer to retrieve data because time would be wasted going through duplicate rows. Also, updating data would be an issue; if a user changes their Twitter handle, we would need to find every duplicate and update it.

Repetitive data is a problem. We can fix this problem by splitting Table 1 into separate tables. Let’s proceed with first resolving the repetition across columns issue.


Step 2: Remove Repetitive Data Across Columns

As noted above, the “username” and “following_username” columns in Table 1 are repetitive. This repetition occurred because I was trying to express the follow relationship between users. Let’s improve on Table 1‘s design by splitting it up into two tables: one just for the following relationships and one for the rest of the information.

Fig. 2

Because @Brett_Englebert follows @RealSkipBayless, the following table will express that relationship by storing @Brett_Englebert as the “from_user” and @RealSkipBayless as the “to_user.” Let’s go ahead and split Table 1 into these two tables:

Table 2: The following table

from_userto_user_DreamLeadScootmedia_DreamLeadMetiersInternetGunnarSvalanderkloutGunnarSvalanderzillowGEsoftwareDayJobDocGEsoftwarebyoskoadrianburchCindyCrawfordadrianburchArjantimAndyRyderMichaelDellAndyRyderYahooBrett_EnglebertRealSkipBaylessBrett_EnglebertstephenasmithNimbusDatadellock6NimbusDatarohitkilamSSWUGorgdrsqlSSWUGorgsteam_games

Table 3: The users table

full_nameusernametextcreated_at“Boris Hadjur”“_DreamLead”“What do you think about #emailing #campaigns #traffic in #USA? Is it a good market nowadays? do you have #databases?”“Tue, 12 Feb 2013 08:43:09 +0000″“Gunnar Svalander”“GunnarSvalander”“Bill Gates Talks Databases, Free Software on Reddit http://t.co/ShX4hZlA #billgates #databases”“Tue, 12 Feb 2013 07:31:06 +0000″“GE Software”“GEsoftware”“RT @KirkDBorne: Readings in #Databases: excellent reading list, many categories: http://t.co/S6RBUNxq  via @rxin Fascinating.”“Tue, 12 Feb 2013 07:30:24 +0000″“Adrian Burch”“adrianburch”“RT @tisakovich: @NimbusData at the @Barclays Big Data conference in San Francisco today, talking #virtualization, #databases, and #flash memory.”“Tue, 12 Feb 2013 06:58:22 +0000″“Andy Ryder”“AndyRyder5″“http://t.co/D3KOJIvF article about Madden 2013 using AI to prodict the super bowl #databases #bus311″“Tue, 12 Feb 2013 05:29:41 +0000″“Andy Ryder”“AndyRyder5″“http://t.co/rBhBXjma an article about privacy settings and facebook #databases #bus311″“Tue, 12 Feb 2013 05:24:17 +0000″“Brett Englebert”“Brett_Englebert”“#BUS311 University of Minnesota’s NCFPD is creating #databases to prevent “food fraud.” http://t.co/0LsAbKqJ”“Tue, 12 Feb 2013 01:49:19 +0000″Brett Englebert“Brett_Englebert”“#BUS311 companies might be protecting their production #databases, but what about their backup files? http://t.co/okJjV3Bm”“Tue, 12 Feb 2013 01:31:52 +0000″“Nimbus Data Systems”“NimbusData”“@NimbusData CEO @tisakovich @BarclaysOnline Big Data conference in San Francisco today, talking #virtualization, #databases,& #flash memory”“Mon, 11 Feb 2013 23:15:05 +0000″“SSWUG.ORG”“SSWUGorg”“Don’t forget to sign up for our FREE expo this Friday: #Databases, #BI, and #Sharepoint: What You Need to Know! http://t.co/Ijrqrz29″“Mon, 11 Feb 2013 22:15:37 +0000″

This is looking better. Now in the users table (Table 3), there is only one column with Twitter handles. In the following table (Table 2), the is only one Twitter handle per field in the “to_user” column.

Edgar F. Codd, the computer scientist who layed down the theoretical basis of relational databases, called this step of removing repetitive data across columns the first normal form (1NF).


Step 3: Remove Repetitive Data Across Rows

Now that we’ve fixed repetions across columns, we need to fix repetitions across rows. Since the users @AndyRyder5 and @Brett_Englebert each tweeted twice, their information is duplicated in the users table (Table 3). This indicates that we need to pull out the tweets and place them in their own table.

Fig. 3

As before, “text” stores the tweet itself. Since the “created_at” column stores the timestamp of the tweet, it makes sense to pull it into this table as well. I also include a reference to the “username” column so we know who published the tweet. Here is the result of placing the tweets in their own table:

Table 4: The tweets table

textcreated_atusername“What do you think about #emailing #campaigns #traffic in #USA? Is it a good market nowadays? do you have #databases?”“Tue, 12 Feb 2013 08:43:09 +0000″“_DreamLead”“Bill Gates Talks Databases, Free Software on Reddit http://t.co/ShX4hZlA #billgates #databases”“Tue, 12 Feb 2013 07:31:06 +0000″“GunnarSvalander”“RT @KirkDBorne: Readings in #Databases: excellent reading list, many categories: http://t.co/S6RBUNxq  via @rxin Fascinating.”“Tue, 12 Feb 2013 07:30:24 +0000″“GEsoftware”“RT @tisakovich: @NimbusData at the @Barclays Big Data conference in San Francisco today, talking #virtualization, #databases, and #flash memory.”“Tue, 12 Feb 2013 06:58:22 +0000″“adrianburch”“http://t.co/D3KOJIvF article about Madden 2013 using AI to prodict the super bowl #databases #bus311″“Tue, 12 Feb 2013 05:29:41 +0000″“AndyRyder5″“http://t.co/rBhBXjma an article about privacy settings and facebook #databases #bus311″“Tue, 12 Feb 2013 05:24:17 +0000″“AndyRyder5″“#BUS311 University of Minnesota’s NCFPD is creating #databases to prevent “food fraud.” http://t.co/0LsAbKqJ”“Tue, 12 Feb 2013 01:49:19 +0000″“Brett_Englebert”“#BUS311 companies might be protecting their production #databases, but what about their backup files? http://t.co/okJjV3Bm”“Tue, 12 Feb 2013 01:31:52 +0000″“Brett_Englebert”“@NimbusData CEO @tisakovich @BarclaysOnline Big Data conference in San Francisco today, talking #virtualization, #databases,& #flash memory”“Mon, 11 Feb 2013 23:15:05 +0000″“NimbusData”“Don’t forget to sign up for our FREE expo this Friday: #Databases, #BI, and #Sharepoint: What You Need to Know! http://t.co/Ijrqrz29″“Mon, 11 Feb 2013 22:15:37 +0000″“SSWUGorg”

Table 5: The users table

full_nameusername“Boris Hadjur”“_DreamLead”“Gunnar Svalander”“GunnarSvalander”“GE Software”“GEsoftware”“Adrian Burch”“adrianburch”“Andy Ryder”“AndyRyder5″“Brett Englebert”“Brett_Englebert”“Nimbus Data Systems”“NimbusData”“SSWUG.ORG”“SSWUGorg”

After the split, the users table (Table 5) has unique rows for users and their Twitter handles.

Edgar F. Codd called this step of removing repetitive data across rows the second normal form (1NF).


Step 4: Linking Tables with Keys

Data can be inserted, retrieved, updated, and deleted from a table.

So far, Table 1 has been split into three new tables: following (Table 2), tweets (Table 4), and users (Table 5). But how is this useful? Repetitive data has been removed, but now the data is spread out across three independent tables. In order to retrieve the data, we need to draw meaningful links between the tables. This way we can express queries like, “what a user has tweeted and who a user is following”.

The way to draw links between tables is to first give each row in a table a unique identifier, termed a primary key, and then reference that primary key in the other table to which you want to link.

We’ve actually already done this in users (Table 5) and tweets (Table 4). In users, the primary key is the “username” column because no two users will have the same Twitter handle. In tweets, we reference this key in the “username” column so we know who tweeted what. Since it is a reference, the “username” column in tweets is called a foreign key. In this way, the “username” key links the users and tweets tables together.

Is the “username” column the best idea for a primary key for the users table?

On one hand, it’s a natural key — it makes sense to search using a Twitter handle instead of assigning each user some numerical ID and searching on that. On the other hand, what if a user wants to change his Twitter handle? That could cause errors if the primary key and all referencing foreign keys aren’t updated accurately, errors that could be avoided if a constant numerical ID was used. Ultimately the choice depends on your system. If you want to give your users the ability to change their username, it’s better to add a numerical auto-incrementing “id” column to users and use that as the primary key. Otherwise, “username” should do just fine. I’ll continue to use “username” as the primary key for users

Let’s move on to tweets (Table 4). A primary key should uniquely identify each row, so what should the primary key be here? The “created_at” field won’t work because if two users tweet at the exact same time, their tweets would have an identical timestamp. The “text” has the same problem in that if two users both tweet “Hello world,” we couldn’t distinguish between the rows. The “username” column is the foreign key that defines the link with the users so let’s not mess with that. Since the other columns are not good candidates, it makes sense here to add a numerical auto-incrementing “id” column and use that as the primary key.

Table 6: The tweets table with an “id” column

idtextcreated_atusername1“What do you think about #emailing #campaigns #traffic in #USA? Is it a good market nowadays? do you have #databases?”“Tue, 12 Feb 2013 08:43:09 +0000″“_DreamLead”2“Bill Gates Talks Databases, Free Software on Reddit http://t.co/ShX4hZlA #billgates #databases”“Tue, 12 Feb 2013 07:31:06 +0000″“GunnarSvalander”3“RT @KirkDBorne: Readings in #Databases: excellent reading list, many categories: http://t.co/S6RBUNxq  via @rxin Fascinating.”“Tue, 12 Feb 2013 07:30:24 +0000″“GEsoftware”4“RT @tisakovich: @NimbusData at the @Barclays Big Data conference in San Francisco today, talking #virtualization, #databases, and #flash memory.”“Tue, 12 Feb 2013 06:58:22 +0000″“adrianburch”5“http://t.co/D3KOJIvF article about Madden 2013 using AI to prodict the super bowl #databases #bus311″“Tue, 12 Feb 2013 05:29:41 +0000″“AndyRyder5″6“http://t.co/rBhBXjma an article about privacy settings and facebook #databases #bus311″“Tue, 12 Feb 2013 05:24:17 +0000″“AndyRyder5″7“#BUS311 University of Minnesota’s NCFPD is creating #databases to prevent “food fraud.” http://t.co/0LsAbKqJ”“Tue, 12 Feb 2013 01:49:19 +0000″“Brett_Englebert”8“#BUS311 companies might be protecting their production #databases, but what about their backup files? http://t.co/okJjV3Bm”“Tue, 12 Feb 2013 01:31:52 +0000″“Brett_Englebert”9“@NimbusData CEO @tisakovich @BarclaysOnline Big Data conference in San Francisco today, talking #virtualization, #databases,& #flash memory”“Mon, 11 Feb 2013 23:15:05 +0000″“NimbusData”10“Don’t forget to sign up for our FREE expo this Friday: #Databases, #BI, and #Sharepoint: What You Need to Know! http://t.co/Ijrqrz29″“Mon, 11 Feb 2013 22:15:37 +0000″“SSWUGorg”

Finally, let’s add a primary key to the following table. In this table, neither the “from_user” column nor the “to_user” column uniquely identifies each row on its own. However “from_user” and “to_user” together do, since they represent a single relationship. A primary key can be defined on more than one column, so we’ll use both these columns as the primary key for the following table.

As for the foreign key, “from_user” and “to_user” are each foreign keys since they can be used to define a link with the users table. If we query for a Twitter handle on the “from_user” column, we’ll get all the users he follows. Correspondingly, if we query for a Twitter handle on the “to_user” column, we’ll get all the users following him.

We’ve accomplished a lot so far. We removed repetitions across columns and rows by separating data into three different tables, and then we chose meaningful primary keys to link the tables together. This entire process is called normalization and its output is data that is cleanly organized according to the relational model. The consequence of this organization is that rows will appear in the database only once moving forward, which in turn make the CRUD operations easier.

Fig. 4 diagrams the finalized database schema. The three tables are linked together and the the primary keys are highlighted.

Fig. 4


Relational Database Management Systems

There are small variations in SQL between each RDBMS vendor, termed SQL dialects.

Now that we know how to design a relational database, how do we actually implement one? Relational database management systems (RDBMS) are software that let you create and use relational databases. There are several commercial and open source vendors to choose from. On the commercial side, Oracle Database, IBM DB2, and Microsoft SQL Server are three well known solutions. On the free and open source side, MySQL, SQLite, and PostgreSQL are three widely used solutions.

MySQL is used at just about every Internet company you have heard of. In context of this article, Twitter uses MySQL to store their users’ tweets.

SQLite is common in embedded systems. iOS and Android let developers use SQLite to manage their app’s private database. Google Chrome uses SQLite to store your browsing history, cookies, and your thumbnails on the “Most visited” page.

PostgreSQL is also a widely used RDBMS. Its PostGIS extension supplements PostgreSQL with geospatial functions that make it useful for mapping applications. A notable user of PostgreSQL is OpenStreetMap.


Structured Query Language (SQL)

Once you’ve downloaded and set up an RDBMS on your system, the next step is to create a database and tables inside of it in order to insert and manage your relational data. The way you do this is with Structured Query Language (SQL), which is the standard language for working with RDBMSs.

Here’s a brief overview of common SQL statements that are relevant to the example Twitter data above. I recommend you check out the SQL Cookbook for a more complete application-driven list of SQL queries.

  • Create a database, named “development”
      CREATE DATABASE development;
      
  • Create a table named “users”
      CREATE TABLE users (
        full_name VARCHAR(100),
        username VARCHAR(100)
      );
      

    RDBMSs require that each column in a table is given a data type. Here I’ve assigned the “full_name” and “username” columns the data type VARCHAR which is a string that can vary in width. I’ve arbitrarily set a max length of 100. A full list of data types can be found here.

  • Insert a record (the Create operation in CRUD)
      INSERT INTO users (full_name, username)
      VALUES ("Boris Hadjur", "_DreamLead");
      
  • Retrieve all tweets belonging to @_DreamLead (the Retrieve operation in CRUD)
      SELECT text, created_at FROM tweets WHERE username="_DreamLead";
      
  • Update a user’s name (the Update operation in CRUD)
      UPDATE users
      SET full_name="Boris H"
      WHERE username="_DreamLead";
      
  • Delete a user (the Delete operation in CRUD)
  •   DELETE FROM users
      WHERE username="_DreamLead";
      

SQL is pretty similar to regular English sentences. There are small variations in SQL between each RDBMS vendor, termed SQL dialects, but the differences are not dramatic enough that you can’t easily transfer your SQL knowledge from one to the other.


Conclusion

In this article, we learned how to design a relational database. We took a collection of data and organizing it into related tables. We also briefly looked at RDBMS solutions and SQL. So get started by downloading an RDBMS and normalizing some of your data into a relational database today.

Preview Image Source: FindIcons.com/Barry Mieny

August 08 2012

18:45

How Web Developers Can Use Alfred

In this screencast, we’ll use Alfred to write an extension that assists us when creating new projects. As you’ll find, Alfred makes the process of executing predefined shell scripts a cinch.

Choose 720p for the clearest picture.

Final Bash Script

# Copy our Backbone template directory over
cd ~/Desktop && mkdir {query} && cd {query}

# Begin with our template
cp -R ~/Templates/Backbone/ .

# Pull in Backbone
curl http://backbonejs.org/backbone-min.js > assets/js/libs/backbone.js

# Pull in Underscore
curl http://underscorejs.org/underscore-min.js > assets/js/libs/underscore.js

# Pull in jQuery
curl http://code.jquery.com/jquery.js > assets/js/libs/jquery.js

# Open our new project in Sublime Text
/Applications/Sublime Text 2.app/Contents/SharedSupport/bin/subl .
Final Script


April 19 2012

18:48

Lightning Fast Folder and File Creation in Sublime Text 2

I’m frequently asked about how I’m able to create new directory structures and files so quickly in Sublime Text 2. Well the answer is that this functionality is not offered natively; instead, I use a helpful plugin. I’ll demonstrate it in this video.

Choose 720p for the clearest video.

Closing Thoughts

It’s worth mentioning that, toward the end of the video, I assign a custom key-binding: n-n. Please note that this binding assumes that you’re working in Vintage (Vim) mode. To trigger it, you’d exit to command mode – by pressing Esc – and then type n-n.

If you’re not a Vintage user, you can stick with the default key-binding that comes with the plugin.

Show Links


March 06 2012

20:59

Sexy Code Snippet Management With Gists

I’ve always struggled to find the perfect code snippet management tool…until now (and it’s free). I’ll show you my preferred workflow, using GitHub Gists and Sublime Text 2.

View at 720p for optimal viewing.

Summary

So that’s the system I prefer these days; do you have a better method?


February 25 2012

20:49

Essential Sublime Text 2 Plugins and Extensions

Sublime Text 2 is a relatively new code editor that I’ve been trying out for a while now. While it’s still in public beta, it already offers a great mix of features and performance that has convinced me to switch from my trusted Komodo.

While I really do love the features available out of the box, as with most things in life, there is always room for more. With Sublime Text 2 being as extensible as it is, a big ecosystem has sprouted around it, catering to most of your web development needs, be they actually useful or catering to your whimsy. To that effect, today I’d like to share some of the plugins and extensions that I’ve found quite useful. While not all of them may appeal to you, I’m sure you’ll a find a gem or two that will absolutely ease your workflow!


Zen Coding

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

Zen Coding is an editor plugin for high-speed HTML coding and editing. The core of this plugin is a powerful abbreviation engine which allows you to expand expressions—similar to CSS selectors—into HTML code.


jQuery Package for Sublime Text

And where will all us be without jQuery? This is a Sublime Text bundle to help with jQuery functions.


Sublime Prefixr

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

A plugin that runs CSS through the Prefixr API, written by our very own Jeffrey Way, for Sublime Text 2.


JS Format

JsFormat is a javascipt formatting plugin for Sublime Text 2. It uses the commandline/python-module javascript formatter from JS Beautifier to format the selected text, or the entire file if there is no selection.


SublimeLinter

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

SublimeLinter is a plugin that supports “lint” programs (known as “linters”). SublimeLinter highlights lines of code the linter deems to contain (potential) errors. It also supports highlighting special annotations so that they can be quickly located.


Placeholders

I always find inserting placeholder, or filler, content to be a quite tedious affair. With this plugin, you can insert placeholder content and HTML in a cinch!


Sublime Alignment

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

I’m quite a stickler for properly formatted code. One thing to get right is lining up all those darn variable assignment so they look all organized and neat. With this plugin, all it takes is the press of key. A simple key-binding allows you align multi-line and multiple selections.


Clipboard History

Tired of having to swap out your clipboard’s contents during a marathon hackathon? Keep a history of your clipboard items with this plugin and paste away as needed.


SublimeREPL

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

SublimeREPL lets you run your favorite interpreter inside a Sublime buffer. Languages supported include Python and Ruby.


DetectSyntax

DetectSyntax is a plugin for Sublime Text 2 that allows you to detect the syntax of files that might not otherwise be detected properly. This is specially helpful when you run into custom file formats — files used in templating is an excellent example.


Nettuts Fetch

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

This plugin automatically pulls in the latest copy of a file, simply by typing a keyboard shortcut. It’ll perform a curl request to your specified URL and allow you to rest assured that, for all new projects, you’re using the latest copy of a particular asset.


JsMinifier

It’s a good practice to always minify your files during deploying to a production server. And this plugin will swiftly automate the process by minifying your JavaScript using the Google Closure compiler.


Sublime CodeIntel

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

SublimeCodeIntel is a code intelligence plugin ported from Open Komodo Editor to Sublime Text 2. It shows autocomplete information with the available modules in real time as well as display information about the current function in the status bar. Nifty!


Tag

This is a great plugin when you’re working with a lot of markup. Tag is a collection of packages about, predictably, tags, mixed together in an effort to provide a single package with utilities to work with tags. Close a tag on a slash and tag indenting? Sign me up!


Bracket Highlighter

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

This plugin collection includes plugins to fold your code according to brackts, cycle through selecting tags and many more.


Case Conversion

Have a messy co-worker who completely ignores naming conventions? This plugin should save you a good chunk of time. Case conversions converts the current word between three of the most commonly used conventions.


Stackoverflow Search

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

StackOverflow is an absolute life saver — I can’t count the sheer number of times it has saved my skin. This plugin lets you do a search on SO directly from your editor.


Sublime Guard

Remember Jeffrey using a gem called Guard in his super useful Rails tutorial? Well, this plugin provides a seamless interface for controlling Guard and viewing Guard output within Sublime Text 2.


Git

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

A nifty little plugin that integrates Git and Sublime Text and implements most of the commands that you’d use in real life. Diff viewing inside ST2 is a great time saver!


Sublime Change Quotes

This is one for the OCD among us. This plugin converts single to double or double to single quotes whilst attempting to preserve correct escaping.


Hex to HSL

Nettuts+ -- Essential Sublime Text 2 Plugins and Extensions

Tired of constantly having to manually convert your colors’ hexcodes to HSL? This plugin will automatically do it for you with the press of a button. Well, ok, three buttons. [Shift+Ctrl+U]


That’s a Wrap!

Well, that’s about all the plugins and extensions that I use on a daily basis. Considering the teeming ecosystem that Sublime Text 2 boasts, I’m sure I’m missing a ton here. Let us know in the comments and thank you so much for reading!


February 16 2012

14:47

How to Customize Your Command Prompt

Lately, I’ve been getting this question a lot: “how did you get your terminal to look the way it does?” If you’ve noticed my terminal and are curious about how I set it up, this is the tutorial for you! Of course, what you learn here will be enough to get you started on creating your own custom command prompt, as well!


Before we get started, I want to make something extremely clear. I’m certainly a command line enthusiast, but I’m not at all a command line pro. I feel at home in a terminal, but I’m a far cry from knowing everything. So here’s the deal: I’m going to show you how I’ve set up my terminal, but that doesn’t mean I’ll be able to explain every single line of code we’ll look at. There are some things here that are the way they are just because that’s what works … and I don’t always 100% know why.

With that disclaimer out of the way, let’s see what we’re doing.


Meeting the Finished Product

Here’s what my command prompt looks like:

If you’re not sure what you’re looking at there, let me explain:

  • In the turquoise, we have the name of the computer; in my case, that’s mothership. That’s followed by a colon.
  • Next, we have the working directory, in a yellow-orange.
  • If we’re in a git repository, we have some info on that next. The main thing here is the branch name (master or tilt_in_post_class in the screenshot). Also, if the working directory is clean, that text appears in green; otherwise, it appears in red.
  • Finally, we have a battery indicator. If the ten triangles are all green and filled in, things are powered up. As my battery empties, the triangles will empty and eventually turn red. Of course, if you aren’t on a laptop like I am, this won’t be as useful to you.

Getting the Environment Ready

Let’s now get a few preliminary pieces together, before actually writing some shell scripts.

First, there’s the colour scheme. You might recognize it as Ethan Schoonover’s Solarized colour scheme. It’s pretty amazing, and I’ve used it on both the terminal and in Vim ever since I discovered it. If you want to use it in the Terminal, you’ll have to install the theme. The Terminal in Snow Leopard didn’t support xterm-256color, so you’ll have to follow some special directions on the Solarized GitHub page to get that working if you’re still on that OS.

If you’ve moved on to Lion, you can just install the .terminal files you’ll find in the xterm-256color folder. Once you have those installed (just double-click them), you should be able to select the one you want in the Terminal preferences. Don’t forget to set it as the default scheme.

The next thing to know is that I’m not using the default bash shell in my Terminal. Instead, I’ve switched to zsh, which is basically much bash-compatible, but has a few nice additions, like better tab completion. Here’s how to do it: open the Mac system preferences and go to “Users & Groups.” Unlock the pane by clicking the lock at the bottom and entering your password. Then, right-click on your user in the list and choose “Advanced Options.” In the “Login Shell” field, switch from /bin/bash to /bin/zsh. It’s that simple.

Fonts

Next step: get the right font. I’m using Inconsolata at 15pt. It’s a free monospace font that I love staring at all day long. (Besides using it in the Terminal, I use it in Vim and TextEdit.) You can set your default font from within the Terminal preferences, right where you choose the colour scheme.

Another small thing is the size of your window: Open Terminal Preferences > Settings and click the Window tab; part-way down, you can choose the number of columns and rows you want; I use 130 columns by 30 rows.

Battery

Remember the battery level indicator? Well, that requires a little script from developer Steve Losh; simply copy that into a file and save it as a python file; since ~/bin is in my terminal PATH, I’ve saved the file to ~/bin/batcharge.py. As he notes, this script will only work on Mac OS X, so if you’re running zsh on another system, you’ll have to leave this part out.

Zsh

Last, but most certainly not least, there’s oh-my-zsh. According to the Github repo, this is just “A handful of functions, auto-complete helpers, and stuff that makes you shout ‘OH MY ZSHELL!’”

Why use it? For me, I decided to give it a try at one point and I’ve left it installed. If you use the terminal a lot, poke around oh-my-zsh for a bit when you have some time. You might be surprised at what you’ll find. Installing oh-my-zsh is fairly simple: just follow the setup instructions in the README; they’re pretty straight-forward.

Now we have all the necessary parts in place. We’re ready to actually start creating our custom terminal.


Creating the Files

When you installed oh-my-zsh, it was installed to ~/.oh-my-zsh. Pop that open. You’ll see two folders of note: themes and templates. Inside templates, you’ll find a file called zshrc.zsh-template This is a template for you ~/.zshrc file. If you’ve customized your terminal before, you’ll know that the .bashrc file is where your customizations are stored when you’re using a bash shell. The .zshrc is that same thing, except for the zsh shell. So open up that template file; you don’t have to know what exactly is going on; after all, there are a lot of comments in the file that might not make sense. One thing here is important to use. Notice the line that says this:

ZSH_THEME="robbyrussell"

That’s the name of the theme your terminal is using. Look in the themes folder: you’ll see a robbyrussel.zsh-theme file. What we’re going to do is create a theme of our own, so you can replace that string with the name of our new file. I’ve uncreatively called mine ‘doubleend” because it’s got into on both sides of the terminal.

Any other customizations you want to make to your zsh environment can be made in this file. If you use the terminal a lot, check out oh-my-zsh‘s plugins (in the plugins folder): a ton of useful stuff in there.

Don’t forget to copy to zshrc.zsh-template to your home directory and rename it to .zshrc before you make your changes. Now, in the themes folder, create a file with the theme name you set in your .zshrc file. Make sure you give it the .zsh-theme extension. We’re ready to build our custom theme.


Building the Custom Theme

The most important thing in your theme file is the PROMPT variable. It’s contents is your command prompt. To get the idea of this, just start with this in your theme file:

PROMPT='myPrompt=>'

Open a new Terminal window and you should see this:

All right, let’s get to work. We’re going to have to write several functions, but we’ll start with the PROMPT variable. It might not be noticeable when looking at the terminal, but there are actually three lines to my prompt. The first is a blank line, just to give me some breathing room. The second has all the information, and the third has the arrow. That third line is where you actually type the command. So, here’s our start:

PROMPT='

$reset_color→ '

Yes, you can do multiline strings that easily in shell scripting. But what’s up with $reset_color? That’s a variable that oh-my-zsh defines for us; it resets the colour of the output. This requires a short diversion to discuss how we colour different words in the prompt. You see, there’s a code—a series of characters—that switch the following text to a colour. Obviously, there’s a code for each available colour. Don’t worry, there are other variables for the other colours; you don’t have to learn the codes. By the time we get to the third line, though, we want to reset that to the default text colour; so, we use the $reset_color variable.

If you’re curious about the arrow character, it’s the Unicode rightwards arrow (U+2192, &rarr;). That’s all.

So, now our prompt is looking like this:

Looking svelte. Let’s now add the computer name and working directory. This is all for that second line of our PROMPT variable.

$fg[cyan]%m: $fg[yellow]$(get_pwd) 

We start by setting the text colour to cyan; it appears that we’re getting that colour code from an associative array or hash; while I don’t use it, there’s a $bg hash which changes the background colour instead of the foreground (text) colour.

After setting the colour, we have %m this outputs the name of the computer. After the colon and space, we switch the text colour to yellow. Next, we use the dollar sign and parens to add the output of the function get_pwd. This will output our current working directory, we a bit of a twist. If I’m in the home directory, I don’t want to see /Users/andrew, I want to see ~ instead. So, here’s that function:

function get_pwd() {
	echo "${PWD/$HOME/~}"
}

The function shell is pretty straightforward if you’ve written JavaScript before; identical syntax. I’m not sure where that search-and-replace syntax originated, but that looks pretty similar to the Vim search-and-replace syntax: If PWD includes the text $HOME (a system variable for your home directory), replace it with ~.

Now, here’s what’s down:

Good! Now comes the tricky part. You see, I want to right-align the git information and the battery indicator. Since there’s no way to actually right-align, we have to count the number of characters of text we want, subtract that from the width of the window, and add that spacing. It’s pretty hacky, and the code is pretty messy, but it’s all I’ve been able to find that actually works.

Ready? We insert the spacing with a function that I call get_spacing. So add $(get_spacing) to the end of our second line, so it now looks like this:

$fg[cyan]%m: $fg[yellow]$(get_pwd)$(put_spacing)

Now, that function. Of course, here’s the shell:

function put_spacing() {

}

There are four parts inside. Here’s the first.

local git=$(git_prompt_info)
if [ ${#git} != 0 ]; then
    ((git=${#git} - 10))
else
    git=0
fi

We start by getting the output from the git_prompt_info function and storing it in a local variable, git. Next, if the length of that string is not 0, we reset git so that is now the length of the string minus 10. Otherwise, we reset git to 0. This doesn’t seem to make much sense, until you realize what we’re trying to do here. We want to find out how many character “slots” the git information takes up. The tricky part is that we’re reusing the variable git: first it holds the string, then it holds the number representing the number of characters our git info is long. If git is zero characters long, we set git to 0; if it isn’t (meaning we’re in a git repository), we set git to the number of characters in the string, minus 10. This is because the string character count includes the colour codes, which aren’t actually visible, so they don’t take up width. The double parens? Oh, they’re used for doing math.

We do the very same thing for the battery output:

local bat=$(battery_charge)
if [ ${#bat} != 0 ]; then
    ((bat = ${#bat} - 18))
else
    bat=0
fi

In the third part, we figure out how much spaces we’ll need:

local termwidth
(( termwidth = ${COLUMNS} - 3 - ${#HOST} - ${#$(get_pwd)} - ${bat} - ${git} ))

A bit more math: we start with COLUMNS, which is the number of characters the Terminal is wide. We subtract all the appropriate values (the 3 is for two spaces and a colon), and we end up with the fact that we need termwidth number of spaces between the left and right parts of the prompt.

Now, let’s create a string that’s termwidth number of spaces long:

local spacing=""
for i in {1..$termwidth}; do
    spacing="${spacing} "
done
echo $spacing

A simple for-in loop allows us to create the string; then, we return it.

You can’t really tell that the whitespace has been added, so I’ve added some dummy text so you can see that’s it’s been added.

Next up, the Git info. We add $(git_prompt_info) to the end of prompt line 2; as you know, that’s a function call.

$fg[cyan]%m: $fg[yellow]$(get_pwd)$(put_spacing)$(git_prompt_info)

Notice that we don’t change the colour before loading the Git info: the function will take care of that, because it depends on the repository status.

And here’s the function:

function git_prompt_info() {
	ref=$(git symbolic-ref HEAD 2> /dev/null) || return
	echo "$(parse_git_dirty)$ZSH_THEME_GIT_PROMPT_PREFIX$(current_branch)$ZSH_THEME_GIT_PROMPT_SUFFIX"
}

The first line just checks to see if we’re in a Git repository. If we aren’t we return. If we are, the next line echos out the right info. Notice two things here: first, we’re using two variables: $ZSH_THEME_GIT_PROMPT_PREFIX and $ZSH_THEME_GIT_PROMPT_SUFFIX. I’ll show you how these are defined in a second. The other thing is two other functions that are called. These are provided by oh-my-zsh. The current_branch function just returns the current branch. The parse_git_dirty is more interesting, though. If the current branch is dirty (has uncommitted changes), the function will output the $ZSH_THEME_GIT_PROMPT_DIRTY; otherwise it will output $ZSH_THEME_GIT_PROMPT_CLEAN.

I have these four variables defined like so:

ZSH_THEME_GIT_PROMPT_PREFIX="[git:"
ZSH_THEME_GIT_PROMPT_SUFFIX="]$reset_color"
ZSH_THEME_GIT_PROMPT_DIRTY="$fg[red]+"
ZSH_THEME_GIT_PROMPT_CLEAN="$fg[green]"

Based on these variables, a repo on a clean master branch will output [git:master] in green; a dirty master branch will output +[git:master].

And lastly, we call the battery_charge function:

$fg[cyan]%m: $fg[yellow]$(get_pwd)$(put_spacing)$(git_prompt_info) $(battery_charge)

Here’s the battery_charge function:

function battery_charge() {
    if [ -e ~/bin/batcharge.py ]
    then
        echo `python ~/bin/batcharge.py`
    else
        echo '';
    fi
}

If the file exists, we run that file and echo the output. Notice that we use the backticks around the running of the file (those aren’t single quotes): this allows us to execute a string of code as if it was in the terminal. If the file doesn’t exist, we just echo an empty string.

And with that, we’re done! Here’s what we end with:


Conclusion

Well, that’s what my terminal looks like. I’ve forked the oh-my-zsh project on GitHub and added this theme, so you can find it there. If you’re interested in seeing my other dotfiles, I’ve got them on GitHub too.

However, I’m not done with my command line setup yet. While I haven’t made any changes in a while, I’m thinking of including the current user’s name (because I use this same theme on my server), and also some RVM info. Also, I’m not sure why I have the word git in there; I guess I originally had a setup that worked with multiple version control systems … Anyway, the point of all this is that this is something you’ll continually be tweaking. When I make changes, I’ll be sure to push them to GitHub, so you’ll be able to see them there.

Let me leave you with several links that you’ll find useful in hacking around on the command line:

Have fun!


January 01 2012

16:34

The Tools of Modern Mac-Based Web Development: New on Premium

In this week’s Premium screencast, I provide a brain-dump on all of the various tools I use for web development. Sometimes, simply watching another person work can be incredibly helpful. What tools do they use? How do they work with their code editor? I’ll cover all of these things today.

If you’re not a Tuts+ Premium member, it’s our subscription-based portion of Tuts+, where we provide high quality tutorials, ebooks, and courses on a variety of subjects. Be sure to check it out!


Tools Discussed in the Video


September 21 2011

15:23

CentOS 5: File Server Essentials: New on Premium

What powers a file serving environment? In this tutorial, we’ll begin to learn the skills required to install an enterprise grade operating system and discover the power and simplicity that makes CentOS the robust and reliable solution trusted by professional throughout the world. No previous experience of this operating system is assumed and during this presentation we will build on the process of a ‘basic server installation’ in order to get you up and running in no time at all.

Become a Premium member to read this tutorial/screencast, as well as hundreds of other advanced tutorials and screencasts from the Tuts+ network.


Before We Begin …

…Some basic requirements and a few assumptions will be made.

  • This tutorial is intended to be an introduction to CentOS during which we will build a typical server installation without a GUI. No prior knowledge or experience of this operating system is assumed but a basic familiarity with the console environment, downloading and burning a CD/DVD image are assumed.
  • In this instance we will be concentrating on the 32bit version using IPv4 but unless otherwise stated you may assume the 64 bit version is similar with very little modification required (you may need to remove some 32bit applications). IPv6 will not be discussed.
  • All administration tasks will be achieved directly via the console (or a secure shell environment) and you will be shown how to configure the operating system, partition your hard disks, install a file-sharing environment, manage users and maintain a firewall. Additional options (including Apache, PHP/Perl, Virtual Hosts, MySQL, BIND etc … ) will be discussed in a future tutorial.
  • CentOS will run on almost any hardware but for the purpose of this tutorial we will be using the computer system described below. Screenshots have been provided (where possible) to support the main body of text together with reasoning and related notes; and as I am from the UK you should be aware that my writing and examples will carry a harmless ‘anglophile’ bias.
  • Throughout this tutorial I have used the text editor ‘nano’ due to its simplicity for new users but do change this to your preferred editor as and when required.
  • This tutorial is considered to be a guide and there is no guarantee that a replication of the following instruction will work for you without making the necessary changes to suit your needs (i.e. computer name, user names, ip addresses etc …) whilst performing additional steps (based on your network topology) that are beyond the scope of this document.

I hope that this tutorial proves to be useful. Members may continue reading at Tuts+ Premium.


Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
(PRO)
No Soup for you

Don't be the product, buy the product!

close
YES, I want to SOUP ●UP for ...