Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

November 18 2013


Client-Side Security Best Practices

Thanks to HTML5, more and more of an applications’ logic is transferred from server-side to client-side. This requires front-end developers to focus more on security. In this article I will show you how to make your apps more secure. I will focus on techniques that you may not have heard about, instead of just telling you that you have to escape HTML data entered in by users.

Don’t Even Think About HTTP

Of course I don’t want you to serve your content with FTP or plain TCP. What I mean is that if you want your users to be safe when using your website, you need to use SSL (HTTPS). And not only for login sites, or valuable information. For all of your content. Otherwise, when someone is accessing your app from a public network, what they see may be malformed by some hacker inside this network. This is called a main-in-the-middle attack:


When you use SSL, all of the data is encrypted before it’s sent, so even if the attacker gets it, he would not be able to modify or capture it. This is by far the most important step in securing your app.

Strict Transport Security

This HTTP header can come in handy if you want to serve your content using only SSL. When it’s issued by the server (or a <meta> tag, but that will allow at least one request to be HTTP), no insecure traffic will come from the browser to your server. It is used like this:

Strict-Transport-Security: max-age=3600; includeSubDomains

The includeSubDomains part is optional, it allows you to declare that you also want all of the sub-domains to be accessed using HTTPS. The max-age option sets how long (in seconds) the pages should be served using SSL. Sadly, only Firefox, Chrome and Opera are supporting Strict Transport Security.

Secure and HttpOnly

Another way to further improve the security on both HTTP and HTTPS are these two cookie attributes: Secure and HttpOnly. The first one allows a cookie to be sent only on SLL connection. The second one may sound as the exact opposite, but it’s not. It’s telling the browser that the cookie can only be accessed using HTTP(S) protocol, so it cannot be stolen using, for example, JavaScript’s document.cookie.

Make XSS Less Harmful With Content Security Policy

Content Security Policy allows you to define the origin of all scripts, images etc. on your site.

If you think your XSS filter will stop all possible XSS attacks check how many ways there are to perform these attacks and think again. Of course securing your app to stop all of these may be a problem and may slow it down, but there is a solution.

It’s called Content Security Policy. It allows you to define the origin of all scripts, images etc. on your site. It also blocks all inline scripts and styles, so even if someone can inject a script tag into a comment or post, the code would not be executed. The CSP is an HTTP header (which can also be set using HTML <meta> tag), which looks like this:

Content-Security-Policy: policy

Where policy is a set of CSP directives. Here are the possible options:

  • script-src – sets acceptable sources of JavaScript code
  • style-src – defines acceptable origins of CSS styles
  • connect-src – specifies the servers the browser can connect to using XHR, WebSockets and EventSource
  • font-src – lists allowed sources of fonts
  • frame-src – defines what origins should be allowed in iframes
  • img-src – sets allowed image sources
  • media-src – lists origins that can serve video and audio files
  • object-src – same as above but for Flash and other plugins

If a directive is not set, the browser assumes that all origins are allowed. This can be changed by setting the default-src option. What you set there will be applied to all unset directives. There is also a sandbox option, which makes the webpage load as an iframe with the sandbox attribute. An example usage of the CSP header would look like this:

Content-Security-Policy: default-src: 'self'; script-src:;

It allows all of the assets to be loaded only from the application’s origin (the 'self' attribute) and also allows you to load scripts from the Google APIs server. There is a lot of flexibility when defining CSP, and when used properly it will greatly improve the security of your webpage.


The thing to remember when using CSP is that, by default, all inline JavaScript will not be executed. This also includes:

  • inline event listeners: like <body onload="main();">
  • all javascript URLs: like <a href="javascript:doTheClick()">

This is because the browser cannot distinguish your inline code from the hacker’s inline code. You will have to replace them by adding event listeners with addEventListener or some framework’s equivalent. This is not a bad thing ultimately, as it forces you to separate your application’s logic from its graphical representation which you should be doing anyway. CSP also (by default) blocks all eval()-ish code, including strings in setInterval/setTimeout and code like new Function('return false').


CSP is available in most of the modern browsers. Firefox, Chrome and Opera (mobile too) use the standard Content-Security-Policy header. Safari (iOS too) and Chrome for Android use the X-WebKit-CSP header. IE10 (with support limited only to the sandbox directive) uses X-Content-Security-Policy. So, thanks to Internet Explorer, you can’t just use only CSP (unless you will use something like Google Chrome Frame), but you can still use it to improve the security on the real browsers and to prepare your app for the future.

Use Cross Origin Resource Sharing Instead of JSONP

JSONP is currently the most used technique to get resources from other servers despite the same-origin policy. Usually, you just create the callback function in your code and pass the name of that function to the URL from which you want to get the data, like this:

function parseData(data) {
<script src=""></script>

But by doing this, you are creating a big security risk. If the server that you are getting data from is compromised, a hacker can add his malicious code and for example, steal your user’s private data, because actually, you are getting JavaScript using this request – and the browser will run all of the code just like with a normal script file.

The solution here is Cross Origin Resource Sharing. It allows your data provider to add a special header in responses so that you can use XHR to retrieve that data, then parse and verify it. This removes the risk of getting malicious code executed on your site.

The implementation requires the provider only to add the following special header in responses:

Access-Control-Allow-Origin: allowed origins

This can be just a few allowed origins separated with spaces, or a wildcard character: * to let every origin request the data.


All current versions of modern browsers support CORS, with the exception of Opera Mini.

Of course, the bigger problem here is that service providers would have to add CORS support, so it’s not completely dependent on the developer.

Sandbox Potentially Harmful Iframes

An iframe with the sandbox attribute will not be able to navigate the window, execute scripts, lock the pointer, show pop-ups or submit forms.

If you are using iframes to load content from external sites, you may want to secure them too. This can be done using the sandbox iframe attribute. An iframe with such an attribute empty (but present) will not be allowed to navigate the window, execute scripts, lock the pointer, show pop-ups or submit forms. The frame will also have a unique origin, so it can’t use localStorage or anything related to the same-origin policy. You can of course allow some of them, if you want, by adding one or more of these values into the attribute:

  • allow-same-origin – the frame will have the same origin as the site, instead of the unique one
  • allow-scripts – the frame will be allowed to execute JavaScript
  • allow-forms – the frame will be able to submit forms
  • allow-pointer-lock – the frame will have access to the Pointer Lock API
  • allow-popups – the frame will be allowed to show pop-ups
  • allow-top-navigation – the frame will be able to navigate the window


The sandbox iframe attribute is supported in all modern browsers, with the exception of Opera Mini.


So that’s it. I hope you’ve learned some new techniques that you can use in your future projects to protect your applications. Thanks to HTML5, we can now do amazing things with our websites, but we have to think about security from the first line of code if we want them to be resistant against attacks.

August 13 2013


Two-Factor Authentication With Duo-Security

In my last article about two-factor authentication, I created a screencast that outlined how to use the Authy two-factor authentication system to easily improve the login security of your website. This time, we’re going to look at another service by Duo Security which offers a solid security API, guides, and pre-built libraries that make it incredibly easy to get up and running quickly.

Just like Authy, you’re going to need an application server to implement Duo since some aspects of it, specifically the secret keys, need to be defined in a non-accessible spot. For this tutorial, I’m using Duo’s ColdFusion library, but they also offer libs for:

  • Python
  • Ruby
  • Classic ASP
  • ASP.Net
  • Java
  • PHP
  • Node.js
  • Perl

From a server-side perspective, they’ve definitely got good coverage.

Getting Your App Ready on Duo

The first thing you’ll need to do is signup for an account. I’m not going to walk you through those steps since it should be self-explanatory, but I will say it was cool to see a free option available so I could create this tutorial at no cost. Yes, Duo charges for their service and honestly that’s a good thing. You want to make sure a service that’s helping you secure your site has funds to keep themselves in business for a bit.

Once you’re signed up, you’ll be prompted to setup a new integration. This just means that you need to setup the specifics of what you’d like to secure. And you’ll see why I say, “what you’d like to secure” when you click on the Integration type dropdown. Duo allows you to setup a variety of security options, not just a website.


For our purposes, we’ll be using the Web SDK” option. The Integration name is any semantic name you’d like to use to identify your site with.


After saving that information, you’re presented with a settings page for your integration. This is where you’ll fine tune how the authentication is supposed to function and get access to your integration, secret keys, and API hostname. These are critical to successfully working with Duo and should not be shared.


In addition, you’ll need to create your own 40-character alphanumeric application key that will not be known to Duo and ensures greater control of your account and application. For example, I generated the following random key for my demo app:


Just note that it must be 40-characters long. Otherwise, you’ll receive an error when you try to use it.

As you look through the settings, most are self-explanatory but there is a section called Policy which allows you to define when a user will be prompted for two-factor authentication. It’s important to choose the best option for your app. From my experience, most sites tend to ask their users if they’d like to opt-in to the enhanced security. Two-factor auth can be cumbersome and some users just don’t want to use it. For this scenario, I'm going to go with the Require Enrollment policy (which will ensure the Duo enrollment process isn't bypassed) and setting a flag in the user's database record when they've opted in. This allows users to login using your normal authentication scheme without being forced to authenticate via Duo.

In reality, that’s really all I needed to setup in the Duo admin panel to make the service available to my app. So let’s start adding in some code.

Adding Duo to My App

I want to reiterate that you’ll need to build server-side code to really make this work and Duo has provided a broad range of libs for you to use.

The code I’m writing is CFML and I’ll be using their ColdFusion component which manages all of the complexities of signing and encrypting my request as well as verifying the return value from the Duo API.

As I mentioned earlier, most two-factor activations are opt-in meaning that a user will go to their account settings, click on a link to turn on the service and go through a process of filling in relevant information to make things work. This generally involves providing the service a cell phone number and validating the settings based on a unique number sent either via text message or a phone call. Duo can offer users either option and also provides their own mobile app that can generate the passcode for users via their phone.

If you look at the screenshot below, you can see how I’ve tried to replicate a simple account screen with a prominent button below it, as a call-to-action for turning on the authentication:


When the user clicks it, a call is made to the Duo component to sign the request via the signRequest() method.

	<cfset session.sigReq = createObject("component","com.duoweb").signRequest( application.IKey, application.SKey, application.AKey, "" ) /> 

To understand what this method does, I’d like to use a quote from the Duo site:

sign_request() performs a HMAC-SHA1 of the username, integration key, and an expiration timestamp, using the integration's secret key as the HMAC key. By generating this server-side and after primary authentication, Duo is assured that the user is indeed authorized to proceed to the secondary stage of authentication.

Basically, it’s creating an encrypted request based on all of the keys for your integration, the unique 40-char application key you created, and the user’s unique username. The end result looks something like this:


The signature gets stored in the variable session.sigReq which is a persistent session-based variable that I can reference later. I check its value to ensure that a valid signature was passed back and if so, I can move on to the next page in the process.


The signature is passed to Duo’s IFRAME which manages both the addition of new users to the service, as well as the validation of existing users. Duo offers a JavaScript library that interacts with the IFRAME to provide the UI to setup users. Looking at the code below, we can see the IFRAME, the reference to the Duo JS lib, and the method call to initialize everything:

<iframe id="duo_iframe" width="100%" height="500" frameborder="0"></iframe>
<script src="js/Duo-Web-v1.bundled.js" type="text/javascript"></script>        
  'host': '',
    'sig_request': '<cfoutput>#session.sigReq#</cfoutput>',
    'post_action': 'success.cfm'

The method call is straightforward, taking three options:

  • The API hostname that was defined for your integration.
  • The signature request that we generated.
  • The URL that Duo will post the results to once it’s done doing its processing.

If you’re confused by this, <cfoutput>#session.sigReq#</cfoutput>, don’t be. It’s just ColdFusion’s way of replacing a variable with its value.

At this point, the user will be presented with the Duo setup screen:


The user will need to enter a phone number and then choose whether they would like to receive their six-digit validation code via voice or text message. I tried both and they worked equally well. Duo does verification on their end to ensure the code being entered is valid.

Next, the user will be presented with a screen to download the Duo mobile app:


This is actually a good thing because having the mobile app will allow the user to get a code even if they have no cell service.

Once their successfully enrolled, they’ll receive the page shown below and asked one more time to validate themselves:


For all intents, all of this process is in Duo’s hands; you’re just waiting for feedback.

That feedback will determine if the user has been properly setup and you’ll need to use the verifyResponse() method for that.

<cfset authUser = createObject("component","com.duoweb").verifyResponse( application.IKey, application.SKey, application.AKey, form.sig_response ) />

Like before, it takes all of the key variables and in this case, receives a response from Duo in the form of a posted variable called sig_response. I’ve referenced it as form.sig_response since that’s how ColdFusion allows access to posted variables.

The verifyResponse() method will take the signed response sent back by Duo and if all is well, will return the user’s username for you to validate against your database. So in my case, I would expect that “” would be returned. Once I've validated it, I then set the flag in the user's database record that would let me know they've opted into two-factor authentication.

That’s it. That’s all you need to do to setup users to activate two-factor authentication. Now let's shift to the login experience.

Logging in With Duo

You might expect something magical to happen from here, but interestingly enough, you can almost reuse the same exact code created for activating a user to allow them to login. I went ahead and created a very basic login page:


The page itself is just HTML markup. The important part is to first determine if the user has opted-in and that happens when you validate their normal site login information. Yes, you should still do your normal login validation of username and password. Duo’s service is complementary to that, not a replacement.

By checking the database record, you should be able to determine if they've opted-in. If they haven't, then you'd only authenticate them using your normal username/password combination. If they have, then you're going to call the signRequest() method, the same one we used when activating a new user:

<cfset session.sigReq = createObject("component","com.duoweb").signRequest( application.IKey, application.SKey, application.AKey, form.username ) /> 

Again, we’re creating an encrypted signature to send to Duo’s API via its IFRAME and JavaScript library. The key thing is that in this scenario, we need to only enforce two-factor authentication if the user has signed up for it. This is why setting the right policy for your integration is important. By setting mine to Require Enrollment and using a database record flag, I can still allow the user to access my site even if they haven’t opted in for two-factor authentication. If the user has opted in, then they’ll be prompted to enter a Duo code to validate their account.

Wrapping up

Increasing the security of one’s site is always a good thing. You want to make sure you protect your users as much as possible and using two-factor authentication is a big step in the right direction.

Duo offers a solid service with incredible ease and flexibility. While I only showed you their Web SDK, they also have a much more flexible API that gives you very granular control over most aspects of the process. While I recommend using the Web SDK, it’s great knowing you have that power at your disposal. Hat’s off to Duo for creating a great service.

Sponsored post

July 25 2013


Two-Factor Auth Using Authy

It feels like every day you read about a new security breach on a website, in many cases involving weak passwords. It’s a known fact that users are notoriously lax when it comes to choosing their passwords, so there needs to be a better way of providing secure logins to your site. That’s where two-factor security comes in. It aims to complement your existing login scheme by also providing an alternative verification method (generally a cellphone) to validate that a user is, who they say they are. It’s a scheme in use by top sites such as Twitter and GMail and has proven to be very reliable in minimizing intrusions via weak passwords.

In this screencast, I’ll introduce you to a great service called Authy which provides a platform to build two-factor authentication right into your site.

Download Video

July 12 2013


OAuth 2.0 – The Good, The Bad & The Ugly

In a world dominated by social media, it’s hard to not come across a client application which you have used to access restricted resources on some other server, for example, you might have used a web-based application (like NY Times) to share an interesting news article on your Facebook wall or tweet about it. Or, you might have used Quora’s iPhone app that accesses your Facebook or Google+ profile and customizes the results based on your profile data, like suggesting to add/invite other users to Quora, based on your friends list. The question is, how do these applications gain access to your Facebook, Twitter or Google+ accounts and how are they able to access your confidential data? Before they can do so, they must present some form of authentication credentials and authorization grants to the resource server.

Introduction to OAuth 2.0

OAuth is often described as a valet key for the web.

Now, it would be really uncool to share your Facebook or Google credentials with any third-party client application that only needs to know about your Friends, as it not only gives the app limitless and undesirable access to your account, but it also has the inherent weakness associated with passwords. This is where OAuth comes into play, as it outlines an access-delegation/authorization framework that can be employed without the need of sharing passwords. For this reason, OAuth is often described as a valet key for the web. It can be thought of as a special key that allows access to limited features and for a limited period of time without giving away full control, just as the valet key for a car allows the parking attendant to drive the car for a short distance, blocking access to the trunk and the on-board cell phone.

However, OAuth is not a new concept, but a standardization and combined wisdom of many well established protocols. Also worth noting is that OAuth is not just limited to social-media applications, but provides a standardized way to share information securely between various kinds of applications that expose their APIs to other applications. OAuth 2.0 has a completely new prose and is not backwards compatible with its predecessor spec. Having said that, it would be good to first cover some basic OAuth2.0 vocabulary before diving into further details.

  • Resource Owner : An entity capable of granting access to a protected resource. Most of the time, it’s an end-user.
  • Client : An application making protected resource requests on behalf of the resource owner and with its authorization. It can be a server-based, mobile (native) or a desktop application.
  • Resource Server : The server hosting the protected resources, capable of accepting and responding to protected resource requests.
  • Authorization Server : The server issuing access grants/tokens to the client after successfully authenticating the resource owner and obtaining authorization.
  • Access Token : Access tokens are credentials presented by the client to the resource server to access protected resources. It’s normally a string consisting of a specific scope, lifetime and other access attributes and it may self contain the authorization information in a verifiable manner.
  • Refresh Token : Although not mandated by the spec, access tokens ideally have an expiration time which can last anywhere from a few minutes to several hours. Once an access token is expired, the client can request the authorization server to issue a new access token using the refresh token issued by the authorization server.

What’s Wrong With OAuth 1.0?

OAuth 1.0′s major drawback was the inherent complexity needed to implement the spec.

Nothing really! Twitter still works perfectly fine with OAuth 1.0 and just started supporting a small portion of the 2.0 spec. OAuth 1.0 was a well thought spec and it allowed exchange of secret information securely without the overhead imposed by SSL. The reason why we needed a revamp was mostly based around the complexity faced in implementing the spec. The following are some areas where OAuth 1.0 failed to impress:

  • Signing Every Request : Having the client generate signatures on every API request and validating them on the server everytime a request is received, proved to be major setback for developers, as they had to parse, encode and sort the parameters before making a request. OAuth 2.0 removed this complexity by simply sending the tokens over SSL, solving the same problem at network level. No signatures are required with OAuth 2.0.
  • Addressing Native Applications : With the evolution of native applications for mobile devices, the web-based flow of OAuth 1.0 seemed inefficient, mandating the use of user-agents like a Web Browser. OAuth 2.0 have accommodated more flows specifically suitable for native applications.
  • Clear Separation of Roles : OAuth 2.0 provides the much needed separation of roles for the authorization server authenticating and authorizing the client, and that of the resource server handling API calls to access restricted resources.

OAuth 2.0 in Depth

Before initiating the protocol, the client must register with the authorization server by providing its client type, its redirection URL (where it wants the authorization server to redirect to after the resource owner grants or rejects the access) and any other information required by the server and in turn, is given a client identifier (client_id) and client secret (client_secret). This process is known as Client Registration. After registering, the client may adopt one of the following flows to interact with the server.

Various OAuth Flows

OAuth 2.0 brings about five new flows to the table and gives developers the flexibility to implement any one of them, depending on the type of client involved:

  • User-Agent Flow : Suitable for clients typically implemented in user-agents (for example, clients running inside a web browser) using a scripting language such as JavaScript. Mostly used by native applications for mobile or desktop, leveraging the embedded or external browser as the user-agent for authorization and it uses the Implicit Grant authorization.
  • Web Server Flow : This makes use of the Authorization Code grant and is a redirection-based flow which requires interaction with the end-user’s user-agent. Thus, it is most suitable for clients which are a part of web-server based applications, that are typically accessed via a web browser.
  • Username and Password Flow : Used only when there is a high trust between the client and the resource owner and when other flows are not viable, as it involves the transfer of the resource owner’s credentials. Examples of clients can be a device operating system or a highly privileged application. This can also be used to migrate existing clients using HTTP Basic or Digest Authentication schemes to OAuth by converting the stored credentials to an access token.
  • Assertion Flow : Your client can present an assertion such as SAML Assertion to the authorization server in exchange for an access token.
  • Client Credentials Flow : OAuth is mainly used for delegated access, but there are cases when the client owns the resource or already has been granted the delegated access outside of a typical OAuth flow. Here you just exchange client credentials for an access token.

Discussing each flow in detail would be out of the scope of this article and I would rather recommend reading the spec for in-depth flow information. But to get a feel for what’s going on, let’s dig deeper into one of the most used and supported flows: The Web Server Flow.

The Web Server Flow

Since this is a redirection-based flow, the client must be able to interact with the resource owner’s user agent (which in most cases is a web browser) and hence is typically suited for a web application. The below diagram is a bird’s eye view of how the end-user (or the resource owner) uses the client application (web-server based application in this case) to authenticate and authorize with the authorization server, in order to access the resources protected by the resource server.


Authenticate & Authorize the Client

The client, on behalf of the resource owner, initiates the flow by redirecting to the authorization endpoint with a response_type parameter as code, a client identifier, which is obtained during client registration, a redirection URL, a requested scope (optional) and a local state (if any). To get a feel for how it really works, here’s a screenshot of how a typical request/response would look like:


This normally presents the resource owner with a web interface, where the owner can authenticate and check what all permissions the client app can use on the owner’s behalf.


Assuming that the resource owner grants access to the client, the authorization server redirects the user-agent back to the client using the redirection URL provided earlier, along with the authorization code as shown by the response below.


Exchange Authorization Code for Tokens

The client then posts to another authorization endpoint and sends the authorization code received in the earlier step, along with the redirection URL, its client identifier and secret, obtained during client registration and a grant_type parameter must be set as authorization_code.


The server then validates the authorization code and verifies that the redirection URL is the same as it was in the earlier step. If successful, the server responds back with an access token and optionally, a refresh token.


Request Restricted Resources Using Access Tokens

The client can now consume the APIs provided by the implementation and can query the resource server for a restricted resource by passing along the access token in the Authorization header of the request. A sample CURL request to Google’s Blogger API to get a blog, given its identifier, would look like this:

	$ curl -H 'Authorization: OAuth ya29.AHES6ZRTj1GNxAby81Es-p_YPWWNBAFRvBYVsYj2HZJfJHU'  

Note that we have added the Access token as an Authorization header in the request and also escaped the token by including it in single quotes, as the token may contain special characters. Keep in mind that escaping the access token is only required when using curl. It results in sending the following request:


The resource server then verifies the passed credentials (access token) and if successful, responds back with the requested information.


These examples are courtesy of OAuth2.0 Playground and are typical to how Google implements the spec. Differences might be observed when trying the same flow with other providers (like Facebook or Salesforce) and this is where interoperability issues creep in, which we discuss a bit later.

Refreshing Access Token

Although not mandated by the spec, but usually the access tokens are short-lived and come with an expiration time. So when an access token is expired, the client sends the refresh token to the authorization server, along with its client identifier and secret, and a grant_type parameter as refresh_token.


The authorization server then replies back with the new value for the access token.


Although a mechanism does exist to revoke the refresh token issued, but normally the refresh token lives forever and must be protected and treated like a secret value.

What’s Wrong With OAuth 2.0?

The Good Stuff

Going by the adoption rate, OAuth 2.0 is definitely an improvement over its arcane predecessor. Instances of developer community faltering while implementing the signatures of 1.0 are not unknown. OAuth 2.0 also provides several new grant types, which can be used to support many use-cases like native applications, but the USP of this spec is its simplicity over the previous version.

The Bad Parts

There are a few loose ends in the specification, as it fails to properly define a few required components or leaves them up to the implementations to decide, such as:

Loose ends in the OAuth 2.0 spec are likely to produce a wide range of non-interoperable implementations.

  • Interoperability: Adding too many extension points in the spec resulted in implementations that are not compatible with each other, what that means is that you cannot hope to write a generic piece of code which uses Endpoint Discovery to know about the endpoints provided by the different implementations and interact with them, rather you would have to write separate pieces of code for Facebook, Google, Salesforce and so on. Even the spec admits this failure as a disclaimer.
  • Short Lived Tokens: The spec does not mandate the lifetime and scope of the issued tokens. The implementation is free to have a token live forever. Although most of the implementations provide us with short-lived access tokens and a refresh token, which can be used to get a fresh access token.
  • Security: The spec just “recommends” the use of SSL/TLS while sending the tokens in plaintext over the wire. Although, every major implementation has made it a requirement to have secure authorization endpoints as well require that the client must have a secure redirection URL, otherwise it will be way too easy for an attacker to eavesdrop on the communication and decipher the tokens.

The Ugly Spat

It took IETF about 31 draft versions and the resignation of the lead author/developer Eran Hammer from the committee to finally publish the spec. Eran sparked a controversy by calling the spec “a bad protocol and a case of death by a thousand cuts”. According to him, the use of bearer tokens (sending tokens over SSL without signing them or any other verification) over the user of signatures (or MAC-Tokens), used in OAuth 1.0 to sign the request, was a bad move and a result of division of interests between the web and enterprise communities.

Conclusive Remarks

The spec surely leaves many extension points out in open, resulting in implementations that introduce their own parameters, in addition to what the spec already defines, and makes sure that implementations from different providers can’t interoperate with each other. But going with the popularity and adoption rate of this framework, with every big player in town (Google, Twitter, Facebook, Salesforce, Foursquare, Github etc.) implementing and tweaking it the way it suits them, OAuth is far from being a failure. In fact, any web application that plans to expose their APIs to other web applications must support some form of authentication and authorization and OAuth fits the bill here.

For Further Reading

May 31 2013


The Fundamentals of Bash Scripting

Shell scripts are widely used in the UNIX world. They’re excellent for speeding up repetitive tasks and simplifying complex execution logic. They can be as simple as a set of commands, or they can orchestrate complex tasks. In this tutorial, we’ll learn more about the Bash scripting language by writing an example script step-by-step.

The Fizz-Buzz Problem

One of the best way to learn about a new language is by example. Let’s start with one.

The Fizz-Buzz problem is a very simple one. It became famous after a programmer, named Imran, used it as an interview test. It turns out that 90-99.5% of the candidates for a programming job are simply unable to write the simplest program. Imran took this simple Fizz-Buzz game and asked the candidates to solve it. Many followed Imran’s example, and, today, it is one of the most asked frequently asked questions for a programming job. If you’re hiring, and need a way to filter through 90% of the candidates, this is a great problem to present.

Here are the rules:

  • Take and print the numbers between 1 and 100.
  • When a number is divisible by 3, print “Fizz” instead of the number.
  • When it is divisible by 5, print “Buzz” instead.
  • When it is divisible both by 3 and 5, print “FizzBuzz”.

That’s all there is to it. I’m sure most of you can already visualize the two or three if statements to solve this. Let’s work through this using the Bash scripting language.


A shebang refers to the combination of the hash and exclamation mark characters: &num;!. The program loader will look for a shebang on the first line of the script, and use the interpreter specified in it. A shebang consists of the following syntax: &num;!interpreter [parameters]. The interpreter is the program that is used to interpret our language. For bash scripting, that would be /bin/bash. For example, if you want to create a script in PHP and run it in console, you’d probably want to use /usr/bin/php (or the path to the PHP executable on your machine) as the interpreter.


Yes, that will actually work! Isn’t it simple? Just be sure to make your file executable first. Once you do, this script will output your PHP information as you would expect.

Tip: To ensure that your script will work on as many systems as possible, you can use /bin/env in the shebang. As such, instead of /bin/bash, you could use /bin/env bash, which will work on systems where the bash executable is not within /bin.

Outputting Text

The output of a script will be equal to, as you might expect, whatever is outputted from your command. However, if we explicitly want to write something to the screen, we can use echo.


echo "Hello World"

Running this script will print “Hello World” in the console.

csaba@csaba ~ $ ./
Hello World
csaba@csaba ~ $

Introducing Variables

As with any programming language, when writing shell scripts, you can use variables.


message="Hello World"
echo $message

This code produces exactly the same “Hello World” message. As you can see, to assign a value to a variable, simply write its name – exclude the dollar sign in front of it. Also, be careful with spaces; there can’t be any spaces between the variable name and the equal sign. So message="Hello" instead of message = 'Hello'

When you wish to use a variable, you can take the value from it just as we did in the echo command. Prepending a $ to the variable’s name will return its value.

Tip: Semicolons aren’t required in bash scripting. You can use them in most cases, but be careful: they may have a different meaning than what you expect.

Printing the Numbers Between 1 and 100

Continuing on with our demo project, we need to cycle through all numbers between 1 and 100. For this, we’ll need to use a for loop.


for number in {1..100}; do
    echo $number

There are several new things worth noting in this example – which by the way, prints all the numbers from 1 to 100, one number at a time.

  • The for syntax in Bash is: for VARIABLE in RANGE; do COMMAND done.
  • The curly braces will transform 1..100 into a range in our example. They’re used in other contexts as well, which we’ll review shortly.
  • do and for are actually two separate commands. If you want to place two commands on a single line, you’ll need to separate them somehow. One way is to use semicolon. Alternatively you could write the code without a semicolon by moving do to the following line.

for number in {1..100}
    echo $number

The First Decision

Now that we know how to print all the numbers between 1 and 100, it’s time to make our first decision.


for number in {1..100}; do
    if [ $((number%3)) -eq 0 ]; then
        echo "Fizz"
        echo $number

This example will output “Fizz” for numbers divisible by 3. Again, we have to deal with a bit of new syntax. Let’s take them one by one.

  • – this is the classic syntax for an if statement in Bash. Of course, the else part is optional – but required for our logic in this case.
  • if COMMAND-RETURN-VALUE; then...if will execute if the return value of the command is zero. Yes, logic in Bash is zero based, meaning that commands that execute successfully exit with a code of 0. If something goes wrong, on the other hand, a positive integer will be returned. To simplify things: anything other than 0 is considered false.
  • Mathematical expressions in Bash are specified by double parentheses. $((number%3)) will return the remaining value of dividing the variable, number, by 3. Please note that we did not use $ inside the parenthesis – only in front of them.

You might be wondering where the command is in our example. Isn’t there just a bracket with an odd expression in it? Well, it turns out that [ is actually an executable command. To play around with this, try out the following commands in your console.

csaba@csaba ~ $ which [
csaba@csaba ~ $ [ 0 -eq 1 ]
csaba@csaba ~ $ echo $?
csaba@csaba ~ $ [ 0 -eq 0 ]
csaba@csaba ~ $ echo $?

Tip: A command's exit value is always returned into the variable, ? (question mark). It is overwritten after each new command's execution.

Checking for Buzz

We're doing well so far. We have "Fizz"; now let's do the "Buzz" part.


for number in {1..100}; do
    if [ $((number%3)) -eq 0 ]; then
        echo "Fizz"
    elif [ $((number%5)) -eq 0 ]; then
        echo "Buzz"
        echo $number

Above, we've introduced another condition for divisibility by 5: the elif statement. This, of course, translates to else if, and will be executed if the command following it returns true (or 0). As you can observe, the conditional statements within [] are usually evaluated with the help of parameters, such as -eq, which stands for "equals."

For the syntax, arg1 OP arg2, OP is one of -eq, -ne, -lt, -le, -gt, or -ge. These arithmetic binary operators return true if arg1 is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to arg2, respectively.arg1 and arg2 may be positive or negative integers. - Bash Manual

When you're attempting to compare strings, you may use the well-known == sign, or even a single equal sign will do. != returns true when the strings are different.

But the Code isn't Quite Correct

So far, the code runs, but the logic is not correct. When the number is divisible by both 3 and 5, our logic will echo only "Fizz." Let's modify our Code to satisfy the last requirement of FizzBuzz.


for number in {1..100}; do
    if [ $((number%3)) -eq 0 ]; then
    if [ $((number%5)) -eq 0 ]; then
    if [ -z $output ]; then
        echo $number
        echo $output;

Again, we've had to make a handful of changes. The most notable one is the introduction of a variable, and then the concatenation of "Buzz" to it, if necessary. Strings in bash are typically defined between double quotes ("). Single quotes are usable as well, but for easier concatenation, doubles are the better choice. Within these double quotes, you can reference variables: some text $variable some other text" will replace $variable with its contents. When you want to concatenate variables with strings without spaces between them, you may prefer to put the variable's name within curly braces. In most cases, like PHP, you're not required to do so, but it helps a lot when it comes to the code's readability.

Tip: You can't compare empty strings. That would return a missing parameter.

Because arguments inside [ ] are treated as parameters, for "[", they must be different from an empty string. So this expression, even though logical, will output an error: [ $output != "" ]. That's why we've used [ -z $output ], which returns true if the string has a length of zero.

Extract Method for Logical Expression

One way to improve our example is to extract into functions the mathematical expression from the if statements, like so:


function isDivisibleBy {
    return $(($1%$2))

for number in {1..100}; do
    if isDivisibleBy $number 3; then
    if isDivisibleBy $number 5; then
    if [ -z $output ]; then
        echo $number
        echo $output;

We took the expressions comparing the rest with zero, and moved them into a function. Even more, we eliminated the comparison with zero, because zero means true for us. We only have to return the value from the mathematical expression - very simple!

Tip: A function's definition must precede its call.

In Bash, you can define a method as function func_name { commands; }. Optionally, there is a second syntax for declaring functions: func_name () { commands; }. So, we can drop the string, function and add "()" after its name. I personally prefer this option, as exemplified in the example above. It's more explicit and resembles traditional programming languages.

You do not need to specify the parameters for a function in Bash. Sending parameters to a function is accomplished by simply enumerating over them after the function call separated by white spaces. Do not place commas or parenthesis in the function call - it won't work.

Received parameters are automatically assigned to variables by number. The first parameter goes to $1, the second to $2, and so on. The special variable, $0 refers the current script's file name.

Let's Play with Parameters


function exampleFunc {
    echo $1
    echo $0
    echo "$@"
    echo "$*"

exampleFunc "one" "two" "three"

This code will produce the following output:

csaba@csaba ~ $ ./
one two thre

Let's analyze the source, line by line.

  • The last line is the function call. We call it with three string parameters.
  • The first line after the shebang is the function definition.
  • The first line in the function outputs the first parameter: "one". So far so simple.
  • The second line outputs current script's file name. Again, very simple.
  • The third line changes the default character separator to the letter, "X". By default, this is " " (a space). That's how Bash knows how the parameters are separated.
  • The fourth line outputs a special variable, $@. It represents all the parameters as a single word, exactly as specified in the function call.
  • The final line outputs another special variable, $*. It represents all the parameters, taken one-by-one and concatenated with the first letter of the IFS variable. That's why the result is oneXtwoXthre.

Returning Strings From Functions

As I noted earlier, functions in Bash can return only integers. As such, writing return "a string" would be invalid code. Still, in many situations, you need more than just a zero or one. We can refactor our FizzBuzz example so that, in the for statement, we will just make a function call.


function isDivisibleBy {
    return $(($1%$2))

function fizzOrBuzz {
    if isDivisibleBy $1 3; then
    if isDivisibleBy $1 5; then
    if [ -z $output ]; then
        echo $1
        echo $output;

for number in {1..100}; do
    fizzOrBuzz $number

Well, this is the first step. We just extracted all the code into a function, called fizzOrBuzz, and then replaced $number with $1. However, all outputting occurs in the fizzOrBuzz function. We want to output from the for loop with an echo statement, so that we can prepend each line with another string. We have to capture the fizzOrBuzz function's output.

for number in {1..100}; do
    echo "-`fizzOrBuzz $number`"
    fizzBuzzer=$(fizzOrBuzz $number)
    echo "-${fizzBuzzer}"

We've updated our for loop just a bit (no other changes). We've now echoed everything twice in two different ways to exemplify the differences between the two solutions to the same problem.

The first solution to capture the output of a function or another command is to use backticks. In 99% of the cases, this will work just fine. You can simply reference a variable within backticks by their names, as we did with $number. The first few lines of the output should now look like:

csaba@csaba ~/Personal/Programming/NetTuts/The Basics of BASH Scripting/Sources $ ./

As you can see, everything is duplicated. Same output.

For the second solution, we've chosen to first assign the return value to a variable. In that assignment, we used $(), which, in this case, forks the script, executes the code, and returns its output.

&semi;, && and ||

Do you remember that we used semicolon here and there? They can be used to execute several commands written on the same line. If you separate them by semicolons, they will just simply be executed.

A more sophisticated case is to use && between two commands. Yes, that's a logical AND; it means that the second command will be executed only if the first one returns true (it exits with 0). This is helpful; we can simplify the if statements into these shorthands:


function isDivisibleBy {
    return $(($1%$2))

function fizzOrBuzz {
    isDivisibleBy $1 3 && output="Fizz"
    isDivisibleBy $1 5 && output="${output}Buzz"
    if [ -z $output ]; then
        echo $1
        echo $output;

for number in {1..100}; do
    echo "-`fizzOrBuzz $number`"

As our function, isDivisibleBy returns a proper return value, we can then use && to set the variable we want. What's after && will be executed only if the condition is true. In the same manner, we can use || (double pipe character) as a logical OR. Here's a quick example below.

csaba@csaba ~ $ echo "bubu" || echo "bibi"
csaba@csaba ~ $ echo false || echo "bibi"
csaba@csaba ~ $

Final Thoughts

So that does it for this tutorial! I hope that you've picked up a handful of new tips and techniques for writing your own Bash scripts. Thanks for reading, and stay tuned for more advanced articles on this subject.

Tags: Other bash

August 29 2012


Improve Your App’s Performance with Memcached

One of the easiest ways to improve your application’s performance is by putting a caching solution in front of your database. In this tutorial, I’ll show you how to use Memcached with Rails, Django, or Drupal.

Memcached is an excellent choice for this problem, given its solid history, simple installation, and active community. It is used by companies big and small, and includes giants, such as Facebook, YouTube, and Twitter. The Memcached site, itself, does a good job of describing Memcached as a “Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.”

In general, database calls are slow.

In general, database calls are slow, since the query takes CPU resources to process and data is (usually) retrieved from disk. On the other hand, an in-memory cache, like Memcached, takes very little CPU resources and data is retrieved from memory instead of disk. The lightened CPU is an effect of Memcached’s design; it’s not queryable, like an SQL database. Instead, it uses key-value pairs to retrieve all data and you cannot retrieve data from Memcached without first knowing its key.

Memcached stores the key-value pairs entirely in memory. This makes retrieval extremely fast, but also makes it so the data is ephemeral. In the event of a crash or reboot, memory is cleared and all key-value pairs need to be rebuilt. There are no built-in high-availability and/or fail-over systems within Memcached. However, it is a distributed system, so data is stored across multiple nodes. If one node is lost, the remaining nodes carry on serving requests and filling in for the missing node.

Installing Memcached

Installing Memcached is a fairly simple process. It can be done through a package manager or by compiling it from source. Depending on your distribution, you may want to compile from source, since the packages tend to fall a bit behind.

# Install on Debian and Ubuntu
apt-get install memcached

# Install on Redhat and Fedora
yum install memcached

# Install on Mac OS X (with Homebrew)
brew install memcached

# Install from Source
tar -zxvf memcached-1.x.x.tar.gz
cd memcached-1.x.x
make && make test
sudo make install

You’ll want to configure Memcached for your specific needs, but, for this example, we’ll just get it running with some basic settings.

memcached -m 512 -c 1024 -p 11211 -d

At this point, you should be up and running with Memcached. Next, we’ll look at how to use it with Rails, Django and Drupal. It should be noted that Memcached is not restricted to being used within a framework. You can use Memcached with many programming languages through one of the many clients available.

Using Memcached with Rails 3

Rails 3 has abstracted the caching system so that you can change the client to your heart’s desire. In Ruby, the preferred Memcached client is Dalli.

# Add Dalli to your Gemfile
gem 'dalli'

# Enable Dalli in config/environments/production.rb:
config.perform_caching = true
config.cache_store = :dalli_store, 'localhost:11211'

In development mode, you will not normally hit Memcached, so either start Rails in production mode with rails server -e production, or add the above lines to your config/environments/development.rb.

The simplest use of the cache is through write/read methods to retrieve data:

Rails.cache.write 'hello', 'world'      #=> true 'hello'                #=> "world"

The most common pattern for Rails caching is using fetch. It will attempt to retrieve the key (in this case, expensive-query) and return the value. If the key does not exist, it will execute the passed block and store the result in the key.

Rails.cache.fetch 'expensive-query' do
  results = Transaction.
    where(':created > orders.created_at', :created =>
# ... more code working with results

In the example above, the problem is cache expiry. (One of the two hard problems in computer science.) An advanced, very robust solution is to use some part of the results in the cache key itself, so that if the results change, then the key is expired automatically.

users =
users.each do |u|
  Rails.cache.fetch "profile/#{}/#{u.updated_at.to_i}" do

Here, we’re using the epoch of updated_at as part of the key, which gives us built in cache expiration. So, if the user.updated_at time changes, we will get a cache miss on the pre-existing profile cache and write out a new one. In this case, we’ll need to update the user’s updated_at time when their profile is updated. That is as simple as adding:

class Profile < ActiveRecord::Base
  belongs_to :user, touch: true

Now, you have self-expiring profiles without any worry about retrieving old data when the user is updated. It’s almost like magic!

Using Memcached with Django

Once you have Memcached installed, it is fairly simple to access with Django. First, you’ll need to install a client library. We’ll use pylibmc.

# Install the pylibmc library
pip install pylibmc

# Configure cache servers and binding
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '',

Your app should be up and running with Memcached now. Like other libraries, you’ll get basic getter and setter methods to access the cache:

cache.set('hello', 'world')
cache.get('hello')             #=> 'world'

You can conditionally set a key if it does not already exist with add. If the key already exists, the new value will be ignored.

cache.set('hello', 'world')
cache.add('hello', 'mundus')
cache.get('hello')              #=> 'world'

From the Python Decorator Library, you can create create a memoized decorator to cache the results of a method call.

import collections
import functools

class memoized(object):
    '''Decorator. Caches a function's return value each time it is called.
    If called later with the same arguments, the cached value is returned
    (not reevaluated).
    def __init__(self, func):
        self.func = func
        self.cache = {}
    def __call__(self, *args):
        if not isinstance(args, collections.Hashable):
            # uncacheable. a list, for instance.
            # better to not cache than blow up.
            return self.func(*args)
        if args in self.cache:
            return self.cache[args]
            value = self.func(*args)
            self.cache[args] = value
            return value
    def __repr__(self):
        '''Return the function's docstring.'''
        return self.func.__doc__
    def __get__(self, obj, objtype):
        '''Support instance methods.'''
        return functools.partial(self.__call__, obj)

def fibonacci(n):
    "Return the nth fibonacci number."
    if n in (0, 1):
        return n
    return fibonacci(n-1) + fibonacci(n-2)

print fibonacci(12)

Decorators can give you the power to take most of the heavy lifting out of caching and cache expiration. Be sure to take a look at the caching examples in the Decorator Library while you are planning your caching system.

Using Memcached with Drupal

Getting started with Memcached in Drupal starts with installing the PHP extension for Memcached.

# Install the Memcached extension
pecl install memcache

    // Configure Memcached in php.ini
    memcache.hash_strategy = consistent
    memcache.default_port = 11211

    // Tell Drupal about Memcached in settings.php
    $conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/';
    $conf['cache_default_class'] = 'MemCacheDrupal';
    $conf['memcache_key_prefix'] = 'app_name';
    $conf['memcache_servers'] = array(
        '' => 'default',
        '' => 'default'

You’ll need to restart your application for all the changes to take effect.

As expected, you’ll get the standard getter and setter methods with the Memcached module. One caveat is that cache_get returns the cache row, so you’ll need to access the serialized data within it.

    cache_set('hello', 'world');
    $cache = cache_get('hello');
    $value = $cache->data;  #=> returns 'world'

And just like that, you’ve got caching in place in Drupal. You can build custom functions to replicate functionality such as cache.fetch in Rails. With a little planning, you can have a robust caching solution that will bring your app’s responsiveness to a new level.

And You’re Done

While a good caching strategy takes time to refine, it shouldn’t stop you from getting started.

Implementing a caching system can be fairly straightforward. With the right configuration, a caching solution can extend the life of your current architecture and make your app feel snappier than it ever has before. While a good caching strategy takes time to refine, it shouldn’t stop you from getting started.

As with any complex system, monitoring is critical. Understanding how your cache is being utilized and where the hotspots are in your data will help you improve your cache performance. Memcached has a quality stats system to help you monitor your cache cluster. You should also use a tool, like New Relic to keep an eye on the balance between cache and database time. As an added bonus, you can get a free ‘Data Nerd’ tshirt when you sign-up and deploy.

August 13 2012


Apache 2 Advanced Configuration on Unix-Like Systems

In a previous tutorial, we took a look at some of the most basic, but important, Apache configuration directives – what they are for, and how to edit them to fit our needs. For a very basic website (perhaps one with just a few static HTML pages), those simple directives might be all you need to know. Chances are, however, you need a more complex website; today we will look at some advanced directives and configuration settings.

Apache’s behavior is controlled through settings and directives applied in plain-text configuration files that usually end with the extension ".conf".

The most important .conf file is httpd.conf, although that depends on your particular installation and Linux distribution (for example, it might be called apache2.conf instead). These configuration files generally apply to the server as a whole, even if the server is hosting multiple websites.

Per-directory configuration files are called ".htaccess" files by default, and are located within the web server’s public document directory. These allow certain directives to be applied to certain directories and their sub-directories, rather than the entire server and all its hosted websites. Though not advised, you can change the name of the .htaccess file to something else using the AccessFileName directive, which can be set in httpd.conf:

# AccessFileName: The name of the file to look for in each  directory
# for additional configuration directives.
AccessFileName .htaccess

Please note that htaccess is not an extension; it is the file’s name. On UNIX-like operating systems, the dot (.) preceding a file name signifies that the file is hidden.

Considering their location in the file system, not all directives can be applied in .htaccess files, as some might simply not be valid. Every line in a configuration file must begin with one of the following:

  • a #, indicating a comment
  • a valid directive
  • a space
  • an empty line

If a line does not begin with one of the above options, Apache will issue an error message instead of starting its HTTP service. So, it is important for you to ensure that the server configuration is valid. If a directive spans over multiple lines, end each line with a back slash () before proceeding to the next directive.

Maintaining Flexibility

The most important .conf file is httpd.conf.

While making your own changes to the default configuration files, it is best to break those changes out into external files (with meaningful names, of course), and include them in the main configuration file, using the include directive. If separating your settings from the server’s default settings is not possible, at least make a habit to comment out old settings before introducing new ones. This will make it easier to roll back to any file version if the need arises, and it also allows you to upgrade to a newer version of the httpd.conf file with minimum hassle. All you need to do is copy the new version in place of the old, and re-insert your includes at the end of the new file (or at least re-apply your old changes after commenting out the default ones in the new file). For example:

# End of default apache configuration

#Beginning of your new configuration and settings

# Include ports listing
Include /etc/apache2/ports.conf

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

As you can see from the example above, you can include a specific file by name, a directory (and all files and sub-directories therein), or multiple files by using wildcards.

Advanced Setup

In my previous tutorial on Apache, you learned some basic directives that control Apache’s behavior. In this tutorial, we’ll look at a few advanced directives, starting with <Directory>.

The <Directory> Directive

The <Directory> directive allows you to specify settings and directives to apply to directories and sub-directories. This gives you the ability to do all sorts of things, such as limiting access to certain directories and files, and turning on or off certain options to certain directories, among other things.

The <Directory> tags take a path and enclose a block of options to be applied to that directory and its sub-directories. Here is an example:

<Directory />
    Options  None
    AllowOverride  None
    Order  Deny,Allow
    Deny  from all

The opening <Directory> tag in this example specifies a path of /, which means the root directory and all its sub-directories and files. The settings defined inside the <Directory> tags apply to the / path (essentially everything in the root).

Note that htaccess is not an extension; it is the file’s name.

The Options directive declares which server features are valid for the specified directory. In this example, no options are valid for the / path. But you could specify any number of options, like allowing symbolic links, allowing the execution of CGI scripts, allowing server-side includes, and many more.

The AllowOverride directive tells the server which settings declared within the <Directory> tags to override if they are reset in the .htaccess file. When the server finds an .htaccess file, it needs to know which directives declared in that file can override earlier configuration directives. If AllowOverride is set to None, as shown in the example above, then no directives are overridden and any re-declarations in the .htaccess files are completely ignored.

If the AllowOverride directive is set to All, then any directive re-declared in the .htaccess file will override earlier configuration directives. AllowOverride can take specific directive types that can be overridden while the rest will be ignored.

Deny and Allow control access to the directory specified within the opening tag, and are prioritized through the Order directive.

In the above example, Order Deny, Allow means all connecting hosts or IP addresses are denied access to the root directory, except those declared as good hosts. Order Allow, Deny on the other hand would mean that all hosts and IPs are to be allowed access to the root directory except those declared as bad or black-listed.

The Deny from all setting declares that access needs to be denied from all hosts. Since it is not followed by a whitelist, no hosts or IPs have access to the root directory (and this is how it should be for security reasons). For demonstration purposes ONLY, the below example demonstrates how to deny access from all hosts except and

<Directory />
    Order  Deny,Allow
    Deny  from all
    Allow  from

Note that the Order directive defines the precedence of the rules. So, first, we deny access from all hosts, and then we only allow access to both and Alternatively, you can specify the two hosts on a separate line like this:

<Directory />
    Order  Deny,Allow
    Deny  from all
    Allow  from
    Allow from

If you want to allow access to all sub-domains of the host (eg:,, and, you can specify a partial domain name to grant access to instead of listing all sub-domains to be allowed. The following example shows you how:

<Directory />
    Order  Deny,Allow
    Deny  from all
    Allow  from

Additionally, if you want to deny access from all except a particular IP address on the local network, the following example illustrates how you could do that:

<Directory />
    Order  Deny,Allow
    Deny  from all
    Allow  from

If you wanted to allow access to all hosts except for a few bad hosts, you could do something like this:

<Directory /usr/local/apache2/htdocs/>
    Order Allow,Deny
    Allow from all
    Deny from

The above example opens the public directory to all connecting clients except for two bad bots and one spam host. Note the use of either the whole domain name or the partial domain name in the host blacklist.

Similar to <Directory> is <DirectoryMatch>. It encloses a group of directives that apply only to the named directory and its sub-directories. Instead of specifying a path, it takes a regular expression as an argument.

Indexes and DirectoryIndex Directives

When you visit a website, you often just type in the domain name without the specifying a page (eg: as opposed to The reason being because the server usually always gives you the default page.

This functionality is governed by the DirectoryIndex directive on the web server; it tells the server the default page to respond with if no file is specified in the URL.

DirectoryIndex can take multiple values (ie: more than one file name), so that when the server encounters a request that doesn’t specify a particular file, the server will go through the many values one by one until it finds a file with the same name in the requested directory.

<Directory /usr/local/apache2/htdocs/>
    DirectoryIndex  index.php index.html

If a client asks for a directory that doesn’t contain any of the default files listed by the DirectoryIndex directive, the server responds with the directory listing, a list of all files and sub-directories contained within that folder. This could potentially be a security risk, as the files and file system structure of the requested directory are exposed. You can avoid this behavior by using the Indexes option to prevent directory listing on the DocumentRoot level (which is the root directory of the server’s htdocs or public files). If you wish to display folder listings for any sub-directory under the DocumentRoot, you can turn on the Indexes option for that particular folder. The following listing demonstrates this:

<Directory /usr/local/apache2/htdocs/>
    Options  -Indexes

<Directory /usr/local/apache2/htdocs/sub-dir1/>
    Options  Indexes

The above example turns off directory listings under the main public HTML folder (in this case /usr/local/apache2/htdocs/) and all sub-directories, and then turns it back on only for the /usr/local/apache2/htdocs/sub-dir1/ sub-directory.

The <Files> Directive

While the <Directory> directive specifies the permissions or restrictions to be applied to a specific directory, the <Files> directive controls the restrictions and permissions for one or multiple files (wildcards must be used within a file name to specify multiple files). Take a look at this listing:

<Files ".htaccess">
    Order  allow,deny
    Deny  from all

This example uses the <Files> directive to prevent .htaccess files from being viewed by web clients. The image below is a screenshot of what the server responds if you try to retrieve the .htaccess file by requesting something like

Similarly, the <FilesMatch> directive limits the scope of the enclosed directives by file name, just as the <Files> directive does. However, it accepts a regular expression as an argument.

The <Location> Directive

The <Location> directive works similar to the <Directory> directive except that it takes a URL as an argument, as opposed to a path to a local physical directory in the file system. This means that <Location> can be used to control content outside the server.

It is highly advised that you don’t use the <Location> directive with local file system locations, because many URLs may map to the same file.

The <LocationMatch> directive also limits the scope of the enclosed directives by URL. Like the other “Match” directives, it accepts a regular expression as an argument.

The <Limit> and <LimitExcept> Directives

The <Limit> directive controls which HTTP methods (eg: GET, POST, etc) are allowed. As seen in the example below, the <Limit> directive denies the use of the POST, PUT, and DELETE methods to all client requests, except for those originating from

    Order Deny,Allow
    Deny from all
    Allow from

The <LimitExcept> directive provides the opposite functionality. It still controls what HTTP methods are allowed, but it does so in an exclusionary manner. In the example below, <LimitExcept> denies access to all client requests using any HTTP methods other than GET and POST.

<LimitExcept GET POST>
    deny  from all

Configuration Structure and Directive Precedence

It is highly advised that you don’t use the <Location> directive with local file system locations.

Apache’s power comes from the ability to extend the server’s capabilities by modules written by other programmers. As such, directives may be set either in the Apache core code or in installed modules. In order to prioritize a directive’s effect and scope, Apache divides its configuration structure into three levels:

  • Server-level configuration
  • Container directives
  • Per-directory configuration

Server-level configuration includes the default directives set for the server as a whole. These directives can then be overridden by per-directory configuration files (.htaccess files), or within container directives (such as the <Directory> and <Files> tags).

Per-directory files are commonly located within (or can be added to) the public directory file structure; the contents of which are available to sub-administrators and developers, and thus, who have the ability to mess with the server’s configuration by adding all kinds directives to various parts of the server. Therefore, the server administrator has the ability to control which directives can be applied within those files, and which directives can or cannot override the default server configuration.


This article was intended for server administrators to provide a reference for more advanced Apache settings and configuration options. As we learned, you can apply these directives either on the server level or a per-directory level. Depending on how and where you set these directives, your server will combine these settings into a unified final configuration.

August 01 2012


Password-Protect Folders in Seconds

In this lesson, I’ll demonstrate how easy it is to password protect folders with Apache. There will often be situations when you don’t need to build a full authentication system from scratch.

Choose 720p for optimal viewing.

July 19 2012


WebGL Essentials: Part I

WebGL is an in-browser 3D renderer based on OpenGL, which lets you display your 3D content directly into an HTML5 page. In this tutorial I will cover all the essentials you need to get started using this framework.


There are a couple things you should know before we get started. WebGL is a JavaScript API that renders 3D content to an HTML5 canvas. It does this by using two scripts that are known in the "3D world" as Shaders. The two shaders are:

  • The vertex shader
  • The fragment shader

Now don’t get too nervous when you hear these names; it’s just a fancy way of saying, "position calculator" and "color chooser" respectively. The fragment shader is the easier one to understand; it simply tells WebGL what color a given point on your model should be. The vertex shader is a little more technical, but basically it converts the points in your 3D models into 2D coordinates. Because all computer monitors are flat 2D surfaces, and when you see 3D objects on your screen, they are merely an illusion of perspective.

If you want to know exactly how this calculation works, you’d need to ask a mathematician, because it uses advanced 4 x 4 matrix multiplications, which are a bit beyond the ‘Essentials’ tutorial. Luckily, you don’t have to know how it works because WebGL will take care of most of it. So let’s get started.

Step 1: Setting Up WebGL

WebGL has a lot of small settings that you have to setup nearly every time you draw something to the screen. In order to save time and make your code neat, I am going to make a JavaScript object that will contain all the ‘behind the scene’ things in a separate file. To get started, create a new file called ‘WebGL.js’ and place the following code inside it:

function WebGL(CID, FSID, VSID){
	var canvas = document.getElementById(CID);
	if(!canvas.getContext("webgl") && !canvas.getContext("experimental-webgl"))
		alert("Your Browser Doesn't Support WebGL");
		this.GL = (canvas.getContext("webgl")) ? canvas.getContext("webgl") : canvas.getContext("experimental-webgl");	
		this.GL.clearColor(1.0, 1.0, 1.0, 1.0); // this is the color 
		this.GL.enable(this.GL.DEPTH_TEST); //Enable Depth Testing
		this.GL.depthFunc(this.GL.LEQUAL); //Set Perspective View
		this.AspectRatio = canvas.width / canvas.height;
		//Load Shaders Here

This constructor function takes in the IDs of the canvas and the two shader objects. First, we get the canvas element and make sure it supports WebGL. If it does, then we assign the WebGL context to a local variable called "GL". The clear color is simply the background color, and it is worth noting that in WebGL most of the parameters go from 0.0 to 1.0 so you would have to divide your rgb values by 255. So in our example 1.0, 1.0, 1.0, 1.0 means a white background with 100% visibility (no transparency). The next two lines tell WebGL to calculate depth and perspective so that an object closer to you will block objects behind it. Finally, we set the aspect ratio which is calculated by dividing the canvas’s width by it’s height.

Before we continue and load the two shaders, let’s write them. I am going to write these in the HTML file where we are going to put the actual canvas element. Create an HTML file, and place the following two script elements just before the closing body tag:

<script id="VertexShader" type="x-shader/x-vertex">
	attribute highp vec3 VertexPosition;
	attribute highp vec2 TextureCoord;
	uniform highp mat4 TransformationMatrix;
	uniform highp mat4 PerspectiveMatrix;
	varying highp vec2 vTextureCoord;
	void main(void) {
		gl_Position = PerspectiveMatrix * TransformationMatrix * vec4(VertexPosition, 1.0);
		vTextureCoord = TextureCoord;

<script id="FragmentShader" type="x-shader/x-fragment"> 
	varying highp vec2 vTextureCoord;
	uniform sampler2D uSampler;
	void main(void) {
		highp vec4 texelColor = texture2D(uSampler, vec2(vTextureCoord.s, vTextureCoord.t));
		gl_FragColor = texelColor;

The vertex shader is created first, and we define two attributes:

  • the vertex position, which is the location in x,y and z coordinates of the current vertex (Point in your Model)
  • the texture coordinate; the location in the texture image that should be assigned to this point

Next, we create variables for the transformation and perspective matrices. These are used to convert the 3D model into a 2D image. The next line creates a shared variable to the fragment shader, and in the main function we calculate the gl_Position (the final 2D position). We then assign the ‘current texture coordinate’ to the shared variable.

In the fragment shader we just take the coordinates we defined in the vertex shader and we ‘sample’ the texture at that coordinate. Basically we are just getting the color in the texture that corresponds to the current point on our geometry.

Now that we have written the shaders, we can go back to loading them in our JS file. So replace the "//Load Shaders Here" with the following code:

var FShader = document.getElementById(FSID);
var VShader = document.getElementById(VSID);

if(!FShader || !VShader)
	alert("Error, Could Not Find Shaders");
	//Load and Compile Fragment Shader
	var Code = LoadShader(FShader);
	FShader = this.GL.createShader(this.GL.FRAGMENT_SHADER);
	this.GL.shaderSource(FShader, Code);
	//Load and Compile Vertex Shader
	Code = LoadShader(VShader);
	VShader = this.GL.createShader(this.GL.VERTEX_SHADER);
	this.GL.shaderSource(VShader, Code);
	//Create The Shader Program
	this.ShaderProgram = this.GL.createProgram();
	this.GL.attachShader(this.ShaderProgram, FShader);
	this.GL.attachShader(this.ShaderProgram, VShader);
	//Link Vertex Position Attribute from Shader
	this.VertexPosition = this.GL.getAttribLocation(this.ShaderProgram, "VertexPosition");
	//Link Texture Coordinate Attribute from Shader
	this.VertexTexture = this.GL.getAttribLocation(this.ShaderProgram, "TextureCoord");

Your textures have to be in even byte sizes or you will get an error…like 2×2, 4×4, 16×16, 32×32…

We first make sure that the shaders exist, and then we move on to loading them one at a time. The process basically gets the shader’s source code, compiles it, and attaches it to the central shader program. There is a function, called LoadShader, that gets the shader code from the HTML file; we will get to that in a second. We use the ‘shader program’ to link the two shaders together, and it give us access to their variables. We store the two attributes we defined in the shaders; so we can input our geometry into them later.

Now let’s look at the LoadShader function, You should put this outside of the WebGL function:

function LoadShader(Script){
	var Code = "";
	var CurrentChild = Script.firstChild;
		if(CurrentChild.nodeType == CurrentChild.TEXT_NODE)
			Code += CurrentChild.textContent;
		CurrentChild = CurrentChild.nextSibling;
	return Code;

It basically just cycles through the shader and collects the source code.

Step 2: The "Simple" Cube

In order to draw objects in WebGL you are going to need the following three arrays:

  • vertices; the points that make up your objects
  • triangles; tells WebGL how to connect the vertices into surfaces
  • texture coordinates; defines how the vertices are mapped on the texture image

This is referred to as UV mapping. For our example let’s create a basic cube. I will split the cube into 4 vertices per side which connect into two triangles. let’s make a variable that will hold a cube’s arrays.

var Cube = {
	Vertices : [ // X, Y, Z Coordinates
		 1.0,  1.0,  -1.0,
		 1.0, -1.0,  -1.0,
		-1.0,  1.0,  -1.0,
		-1.0, -1.0,  -1.0,
		 1.0,  1.0,  1.0,
		 1.0, -1.0,  1.0,
		-1.0,  1.0,  1.0,
		-1.0, -1.0,  1.0,
		 1.0,  1.0,  1.0,
		 1.0, -1.0,  1.0,
		 1.0,  1.0, -1.0,
		 1.0, -1.0, -1.0,
		-1.0,  1.0,  1.0,
		-1.0, -1.0,  1.0,
		-1.0,  1.0, -1.0,
		-1.0, -1.0, -1.0,
		 1.0,  1.0,  1.0,
		-1.0, -1.0,  1.0,
		 1.0, -1.0, -1.0,
		-1.0, -1.0, -1.0,
		 1.0, -1.0,  1.0,
		-1.0, -1.0,  1.0,
		 1.0, -1.0, -1.0,
		-1.0, -1.0, -1.0
	Triangles : [ // Also in groups of threes to define the three points of each triangle
		//The numbers here are the index numbers in the vertex array
		0, 1, 2,
		1, 2, 3,
		4, 5, 6,
		5, 6, 7,
		8, 9, 10,
		9, 10, 11,
		12, 13, 14,
		13, 14, 15,
		16, 17, 18,
		17, 18, 19,
		20, 21, 22,
		21, 22, 23
	Texture : [ //This array is in groups of two, the x and y coordinates (a.k.a U,V) in the texture
		//The numbers go from 0.0 to 1.0, One pair for each vertex
		 1.0, 1.0,
		 1.0, 0.0,
		 0.0, 1.0,
		 0.0, 0.0,
		 0.0, 1.0,
		 0.0, 0.0,
		 1.0, 1.0,
		 1.0, 0.0,
		 1.0, 1.0,
		 1.0, 0.0,
		 0.0, 1.0,
		 0.0, 0.0,
		 0.0, 1.0,
		 0.0, 0.0,
		 1.0, 1.0,
		 1.0, 0.0,
		 1.0, 0.0,
		 1.0, 1.0,
		 0.0, 0.0,
		 0.0, 1.0,
		 0.0, 0.0,
		 0.0, 1.0,
		 1.0, 0.0,
		 1.0, 1.0

It may seem like a lot of data for a simple cube, however, in in part two of this tutorial, I will make a script that will import your 3D models so you don’t have to worry about calculating these.

You may also be wondering why I made 24 points (4 for each side), when there is really only eight total unique points on a cube? I did this because you can only assign one texture coordinate per vertex; so if we would only put in the 8 points, then the whole cube would have to look the same because it would wrap the texture around all the sides that the vertex touches. But this way, each side has it’s own points so we can put a different part of the texture on each side.

We now have this cube variable and are ready to start drawing it. Let’s go back to the WebGL method and add a Draw function.

Step 3: The Draw Function

The procedure for drawing objects in WebGL has a lot of steps; so, it’s a good idea to make a function to simplify the process. The basic idea is to load the three arrays into WebGL buffers. We then connect these buffers to the attributes we defined in the shaders along with the transformation and perspective matrices. Next, we have to load the texture into memory, and, finally, we can call the draw command. So let’s get started.

The following code goes inside the WebGL function:

this.Draw = function(Object, Texture)
    var VertexBuffer = this.GL.createBuffer(); //Create a New Buffer

    //Bind it as The Current Buffer
    this.GL.bindBuffer(this.GL.ARRAY_BUFFER, VertexBuffer);

    // Fill it With the Data
    this.GL.bufferData(this.GL.ARRAY_BUFFER, new Float32Array(Object.Vertices), this.GL.STATIC_DRAW);

    //Connect Buffer To Shader's attribute
    this.GL.vertexAttribPointer(this.VertexPosition, 3, this.GL.FLOAT, false, 0, 0);

    //Repeat For The next Two
    var TextureBuffer = this.GL.createBuffer();
    this.GL.bindBuffer(this.GL.ARRAY_BUFFER, TextureBuffer);
    this.GL.bufferData(this.GL.ARRAY_BUFFER, new Float32Array(Object.Texture), this.GL.STATIC_DRAW);
    this.GL.vertexAttribPointer(this.VertexTexture, 2, this.GL.FLOAT, false, 0, 0);
    var TriangleBuffer = this.GL.createBuffer();
    this.GL.bindBuffer(this.GL.ELEMENT_ARRAY_BUFFER, TriangleBuffer);
    //Generate The Perspective Matrix
    var PerspectiveMatrix = MakePerspective(45, this.AspectRatio, 1, 10000.0);

    var TransformMatrix = MakeTransform(Object);

    //Set slot 0 as the active Texture

    //Load in the Texture To Memory
    this.GL.bindTexture(this.GL.TEXTURE_2D, Texture);

    //Update The Texture Sampler in the fragment shader to use slot 0
    this.GL.uniform1i(this.GL.getUniformLocation(this.ShaderProgram, "uSampler"), 0);

    //Set The Perspective and Transformation Matrices
    var pmatrix = this.GL.getUniformLocation(this.ShaderProgram, "PerspectiveMatrix");
    this.GL.uniformMatrix4fv(pmatrix, false, new Float32Array(PerspectiveMatrix));

    var tmatrix = this.GL.getUniformLocation(this.ShaderProgram, "TransformationMatrix");
    this.GL.uniformMatrix4fv(tmatrix, false, new Float32Array(TransformMatrix));

    //Draw The Triangles
    this.GL.drawElements(this.GL.TRIANGLES, Object.Trinagles.length, this.GL.UNSIGNED_SHORT, 0);

The vertex shader positions, rotates, and scales your object based on the transformation and perspective matrices. We will go more in depth into transformations in the second part of this series.

I have added two functions: MakePerspective() and MakeTransform(). These just generate the necessary 4×4 Matrices for WebGL. The MakePerspective() function accepts the vertical field of view, the aspect ratio, and the nearest and farthest points as arguments. Anything that is closer than 1 unit and farther than 10000 units will not be displayed, but you can edit these values to get the effect you are looking for. Now let’s take a look at these two functions:

function MakePerspective(FOV, AspectRatio, Closest, Farest){
	var YLimit = Closest * Math.tan(FOV * Math.PI / 360);
	var A = -( Farest + Closest ) / ( Farest - Closest );
	var B = -2 * Farest * Closest / ( Farest - Closest );
	var C = (2 * Closest) / ( (YLimit * AspectRatio) * 2 );
	var D =	(2 * Closest) / ( YLimit * 2 );
	return [
		C, 0, 0, 0,
		0, D, 0, 0,
		0, 0, A, -1,
		0, 0, B, 0
function MakeTransform(Object){
	return [
		1, 0, 0, 0,
		0, 1, 0, 0,
		0, 0, 1, 0,
		0, 0, -6, 1

Both these matrices effect the final look of your objects, but the perspective matrix edits your ’3D world’ like the field of view and the visible objects while the transformation matrix edits the individual objects like their rotation scale and position. With this done we are almost ready to draw, all that’s left is a function to convert an image into a WebGL texture.

Step 4: Loading Textures

Loading a texture is a two step process. First we have to load an image like you would in a standard JavaScript application, and then we have to convert it to a WebGL texture. So let’s start with the second part since we are already in the JS file. Add the following at the bottom of the WebGL function right after the Draw command:

this.LoadTexture = function(Img){
	//Create a new Texture and Assign it as the active one
	var TempTex = this.GL.createTexture();
	this.GL.bindTexture(this.GL.TEXTURE_2D, TempTex);  
	//Flip Positive Y (Optional)
	this.GL.pixelStorei(this.GL.UNPACK_FLIP_Y_WEBGL, true);
	//Load in The Image
	this.GL.texImage2D(this.GL.TEXTURE_2D, 0, this.GL.RGBA, this.GL.RGBA, this.GL.UNSIGNED_BYTE, Img);  
	//Setup Scaling properties
	this.GL.texParameteri(this.GL.TEXTURE_2D, this.GL.TEXTURE_MAG_FILTER, this.GL.LINEAR);  
	this.GL.texParameteri(this.GL.TEXTURE_2D, this.GL.TEXTURE_MIN_FILTER, this.GL.LINEAR_MIPMAP_NEAREST);  
	//Unbind the texture and return it.
	this.GL.bindTexture(this.GL.TEXTURE_2D, null);
	return TempTex;

It’s worth noting that your textures have to be in even byte sizes, or you will receive an error; so they have to be dimensions, like 2×2, 4×4, 16×16, 32×32, and so on. I added the line to flip the Y coordinates simply because my 3D application’s Y coordinates were backward, but it will depend on what you are using. This is due to some programs making 0 in the Y axis the top left corner and some applications make it the bottom left corner. The scaling properties that I set just tell WebGL how the image should up-scale and down-scale. You can play around with different options to get different effects, but I thought these worked best.

Now that we are done with the JS file, let’s return to the HTML file and implement all of this.

Step 5: Wrapping It Up

As I mentioned earlier, WebGL renders to a canvas element. That’s all we need in the body section. After adding the canvas element, your html page should look like the following:

		<!-- Include Our WebGL JS file -->
		<script src="WebGL.js" type="text/javascript"></script>
	<body onload="Ready()">  
	  <canvas id="GLCanvas" width="720" height="480">
	    	Your Browser Doesn't Support HTML5's Canvas.  
	<!-- Your Vertex Shader -->
	<!-- Your Fragment Shader -->

It’s a pretty simple page. In the head area I have linked to our JS file. Now let’s go implement our Ready function, which gets called when the page loads:

//This will hold our WebGL variable
var GL; 
//Our finished texture
var Texture;
//This will hold the textures image 
var TextureImage;
function Ready(){
	GL = new WebGL("GLCanvas", "FragmentShader", "VertexShader");
	TextureImage = new Image();
	TextureImage.onload = function(){
		Texture = GL.LoadTexture(TextureImage);
		GL.Draw(Cube, Texture);
	TextureImage.src = "Texture.png";

So we create a new WebGL object and pass in the IDs for the canvas and shaders. Next, we load the texture image. Once loaded, we call the Draw() method with the Cube and the Texture. If you followed along, your screen should have a static cube with a texture on it.

Now even though I said we will cover transformations next time, I can’t just leave you with a static square; it’s not 3D enough. Let’s go back and add a small rotation. In the HTML file, change the onload function to look like so:

TextureImage.onload = function(){
		Texture = GL.LoadTexture(TextureImage);
		setInterval(Update, 33);

This will call a function called Update() every 33 milliseconds which will give us a frame rate of about 30 fps. Here is the update function:

function Update(){
	GL.GL.clear(16384 | 256);
	GL.Draw(GL.Cube, Texture);

This is a fairly simple function; it clears the screen and then draws the updated Cube. Now, let’s go to the JS file to add the rotation code.

Step 6: Adding Some Spin

I’m not going to fully implement transformations, because I’m saving that for next time, but let’s add a rotation around the Y-axis. The first thing to do is add a Rotation variable to our Cube object. This will keep track of the current angle, and allow us to keep incrementing the rotation. So the top of your Cube variable should look like this:

var Cube = {
	Rotation : 0,
	//The Other Three Arrays

Now let’s update the MakeTransform() function to incorporate the rotation:

function MakeTransform(Object){
	var y = Object.Rotation * (Math.PI / 180.0);
	var A = Math.cos(y);
	var B = -1 * Math.sin(y);
	var C = Math.sin(y);
	var D = Math.cos(y);
	Object.Rotation += .3;	
	return [
		A, 0, B, 0,
		0, 1, 0, 0,
		C, 0, D, 0,
		0, 0, -6, 1


And that’s it! In the next tutorial, we will cover loading models and performing transformations. I hope you enjoyed this tutorial; feel free to leave any questions or comments that you might have below.

October 11 2011


Building Static Sites with Jekyll

A full-blown CMS is rarely necessary. Sometimes, you only need to create a light, static website … but you have just enough pages to make the process of copying template files and making cross-site markup changes a chore. Today, I’ll demonstrate a simple solution—Jekyll—that will make creating small websites a breeze.

Step 0: Meeting Jekyll

Jekyll is a simple, blog aware, static site generator.

Jekyll is a simple, blog aware, static site generator. That’s what the site says. But, what exactly does this mean? A static site generator is a program that takes a set of files and generates your site with them. As you’ll see, we’ll be able to use a set of templates, create the content files separately, and then use Jekyll to generate our site. The “blog aware” part means that we could use this to create a blog, or any website that has a series of post-like entries (such as a portfolio). Let’s give it a try!

Step 1: Installing Jekyll

Refer here for more information on Ruby Gems.

We’ll begin by installing Jekyll; it’s a Ruby gem, so doing so should be pretty straightforward.

gem install jekyll # use `sudo` if your setup requires it

Yep: it’s that easy. There are a few more pieces we could install if we are planning on doing a more complex set-up, however, since we’re not, this will do the trick.

Step 2: Creating our First Template

Every file or folder that does not begin with an underscore will be copied to the generated site.

Next, let’s set up the folders for Jekyll. Create a folder, called example-app for this tutorial; we’ll be creating a little portfolio site for, say, a photographer. This is a great example of where Jekyll shines: it’s a small site that won’t be updated too frequently, but is large enough to the point that you don’t want to open every page when you need to make a markup change.

Inside example-app, create a folder called _layouts. Notice the underscore at the beginning of this folder: any folder or file that begin with an underscore will not be part of the site that Jekyll generates. If they have a name that Jekyll recognizes (such as _config.yml or _layouts), their contents will be used in the generation of the site, but the files themselves won’t show up in the site. Remember this: every file or folder that does not begin with an underscore will be copied to the generated site (which, by the way, defaults to the _site sub-folder).

So, let’s create a layout. We’ll start with a general site layout that includes all the “chrome” for our site. Create a new file, called default.html inside the _layouts folder (the name doesn’t matter), and add the following code to it:

<!DOCTYPE html>
  <meta charset=utf-8 />
  <title> {% if page.title %} {{ page.title }} | {% endif %} John Doe, Photographer </title>
  <link rel="stylesheet" href="/css/styles.css" />

  <div id="main">

      <h1> John Doe Photograghy </h1>

    <nav role="navigation">
        <li><a href="/">Home</a></li>
        <li><a href="/portfolio/">Portfolio</a></li>
        <li><a href="/about">About</a></li>
        <li><a href="/contact">Contact</a></li>

    {{ content }}

      <p>@copy; John Doe Photography 2011 | All Rights Reserved. </p>


A couple of things here to keep in mind…

Firstly, Jekyll uses the Liquid template system (by default). This means, anything you can do with Liquid, you can do in a template in Jekyll. For example, in the <title> tag, we’re using both types of Liquid markup: output markup and tag markup. Output markup may output text (if the variable referenced exists), while tag markup doesn’t. Output markup is delimited by double curly-braces, while tag markup is delimited by a the curly brace / percent sign duo.

The next thing to notice above is what is inside the Liquid tags: things like page.title and content. These are variables provided by Jekyll; you can see the list of available template data in the docs. We can also create custom template data, as we’ll review shortly.

Lastly, notice the CSS we’re linking to: create a css folder in the root of your project and throw this bit of styling into a style.css file:

body {
  font: 16px/1.5 verdana, helvetica-neue, helvetica, arial, san-serif;
  background: black;
  color: #ececec;
  padding: 0;
  margin: 0;
ul {
  margin: 0;
  padding: 0;
a {
  color: #ccc;
  text-decoration: none;

a:hover {
  color: #ececec;
  text-decoration: underline;

#main {
  width: 960px;
  margin: 0 auto;
  background: rgba(255, 255, 255, 0.4);
header {
  padding: 0 10px;
  overflow: hidden;
h1 {
  margin: 0;

nav ul, ul.entries {
  list-style-type: none;
nav li a {
  float: left;
  margin: 5px;
.content {
  padding: 10px;

ul.entries li {
  position: relative;
  margin: 20px auto;
  padding: 20px;
  background: #ececec;
  width: 600px;

ul.entries img {
  width: 600px;

ul.entries li h3 {
  position: absolute;
  bottom: -18px;
  left: 17px;
  font-size: 2em;
ul.entries a {
  color: #ececec;
ul.entries a:hover {
  color: #fff;

footer {
  font-size: 0.65em;
  text-align: center;

Also, create an img folder and add an image, named banner.jpg; we’ll be using it shortly. Any image will do; just crop it to 960px by 300px;.

You might be wondering why we’re using the if statement above if the page.title variable just won’t display if it exists? Well, if it does exists, I want to include the vertical bar after it; another way to write that would be like this:

{{ page.title }}{% if page.title %} | {% endif %}

So, how we do use this template? Well, we need to create a page that will use this template. In the root directory of our project, create an index.html file. Here’s the content:

layout: default
<section role="banner">
  <img src="/img/banner.jpg" />

<section class="content">
  Welcome to John Doe Photography! Please, check out my <a href="/portfolio/">Portfolio</a> to see my work.

Here’s the content of our index.html file. Notice what’s at the top of the file: Jekyll calls this YAML front matter. Any file (that doesn’t start with an underscore) that has YAML front matter will be generated by Jekyll before being put in the _site folder (if it has no underscore or YFM, then it will just be copied _site). In this case, the YAML front matter just tells Jekyll what template we want it to use.

Now, open a terminal, cd into your project directory, and run jekyll. You should see something like this:

WARNING: Could not read configuration. Using defaults (and options).
	No such file or directory - /Users/andrew/Desktop/example-app/_config.yml
Building site: /Users/andrew/Desktop/example-app -> /Users/andrew/Desktop/example-app/_site
Successfully generated site: /Users/andrew/Desktop/example-app -> /Users/andrew/Desktop/example-app/_site

Ignore the warning; we’ll come to that shortly. For now, you can see that the site has been built in a freshly-created _site directory. If you open the _site/index.html file in your browser of choice, you should see … a failure. The problem is that our paths (urls and stylesheet) begin with a forward slash. This means we can’t just view them as files, we need to view them on a server. Sure, you could go start up W/MAMP, but why take the trouble? Jekyll has a built in server. So, run jekyll --server, and go to localhost:4000 to see something like image below:

Tutorial Image

If the image above isn’t enough, look at the code of _site/index.html. You’ll see that the template we specified was blended with the content we provided and—voila!—we have our page.

I want to remind you that it’s the YAML front matter that makes this magic happen; if a file doesn’t start with three dashes, one or more lines of properties, and another line of three dashes, the file will just be copied to the _site folder, no generation taking place.

Step 3: Creating a Portfolio Template

Now that we’re comfortable with the basics, let’s create a portfolio for our fictional photographer. Remember how I noted that Jekyll is “blog aware”? Well, we’re going to use this blog-awareness feature to our advantage: instead of posts, we’ll have portfolio entries.

Posts belong in a folder, called _posts, so create that now. The file name pattern for posts must be specific as well: year-month-day-title.ext. Posts — well, any file in your Jekyll site, really — can be either Markdown or HTML.

So let’s make a few posts: remember, these will actually be entries in our portfolio:


layout: portfolio_entry
image: /img/bikes.jpg
title: Bikes, Black and White
Bikes are used by almost everyone in downtown Amsterdam. These ones are chained to a bike rack.


layout: portfolio_entry
title: Wing and a Prayer
image: /img/wing.jpg
The wing of the AirBus I rode to England.


layout: portfolio_entry
title: Stone Bridge
image: /img/bridge.jpg
An old stone bridge in London.


layout: portfolio_entry
title: Road and Curb
image: /img/road.jpg
Bike lanes here are terribly thin.

Pretty simple, eh? Notice how we’re creating a custom YAML front matter field: image. This is the URL to the image for that entry. Sure, we could build the whole entry HTML here in this file, but what if we want to change that? We’d have to return and change it in every entry. This way, we can instead use our portfolio_entry template to render them. What’s that template look like? It’s pretty simple too:


layout: default

<h2 class="content">{{page.title}}</h2>

<img src="{{ page.image }}" />

{{ content }}

If you looked at the template data page, you’ll know that any custom front matter we add will be available under page; so, here, we can access page.image. We’re also using page.title and content (everything after the last three-dash line).

I should mention here that, while the post title is supposed to be available on the post object, I’ve only been able to get it to work on the page object. Whatever works!

Also, notice that we have this template using our default layout. You can nest templates like that, and make your job even easier.

This gives us our entry (post) pages, but what about the main portfolio page? When writing our navigation in our default layout, I noted that we want it as /portfolio/. So, create a folder, called portfolio in the root directory, and open an index.html file within it.

layout: default
title: Portfolio

<section class="content">

  <p>Check out my images below!</p>

<ul class="entries">
  {% for post in site.posts %}

    <a href="{{ post.url }}">
      <img src="{{ post.image }}" />
      <h3>{{ post.title }}</h3>

  {% endfor %}

This is our most complicated piece yet. Remember, this isn’t a template: it’s a “normal” file, but it can still include Liquid tags. We start by setting layout to default, and title to “Portfolio.”

Notice that, in the HTML, we have a Liquid for-in loop. We retrieve all the posts with sites.posts; then, we loop over those posts with for post in site.posts / endfor. If you’ve worked with WordPress, or any other blogging system, you should be familiar with the concept of a loop. That’s all this is! Inside, as you can see, we can get the standard properties, as well as any front matter we defined (like image).

Now if we run jekyll --server to re-generate the site and start the server, localhost:4000/portfolio/ should show this:

Tutorial Image

And here’s an entry page:

Tutorial Image

Great! You’ve created a portfolio. I’m sure you see, as well, how this works for a blog. Let’s now move on to look at some configuration options for Jekyll.

Step 4: Writing a Config File

There’s a plethora of options for Jekyll. It’s great that all of them have really sensible defaults, but if you want to change them, it’s not hard at all.

There are two ways to set options.

  • First, when you run the program on the command line, you can pass parameters. We’ve already seen the --server parameter, which starts a server after generating the site.
  • A different way, and the way we’ll use here, is in a config file, called _config.yml; this is a YAML file, so each line is a key: value pair, just like in the YAML front matter. Jekyll will look for this file before generating site.

So, make an _config.yml file, and let’s check out some of the most common options.

For a complete list of options, review the configuration documentation.

  • auto: Adding auto: true to your config file will keep Jekyll running, watching your project folder for changes and regenerating the site on the fly.
  • source: If your source files are in a different directory than the one you’re running Jekyll from, you’ll want to set that directory with the source property.
  • destination: By default, the destination for your generated site is ./_site. If you’d like something different, set it here.
  • permalink: The permalink is the path to your posts. By default, that’s /year/month/day/title.html. However, you can customize that if you want. Among others, you can use the variables :year, :month, :day, :title, and :categories. :categories comes from the front matter; all the others come from the post file name. Then, you can set permalink to things like /:year/:month/:title/ or /:categories/:title.html. Bonus tip: if you have a permalink property in the post front matter, it will override the site-wide default.
  • exclude: Like I said above, Jekyll won’t generate files in directories starting with an underscore. But, if you have folders that you want it to ignore, but that don’t start with an underscore, you can do it with exclude in your config file.

Step 5: Deploying the Site

So, let’s say you’ve created the site, and want to set it free, out on the world wide web. How do you do that?

There are several ways to accomplish this. Of course, if it’s a small site that you won’t be updating too often, then simply FTP it up to your server; this might be your only option, if you’re using shared hosting.

If you’ve got a VPS or dedicated hosting setup, you can run more automatically. Check out the deployment documentation for a list of good ideas. If you aren’t sure what to do, try following the directions for using the git post-receive hook; I’ve tried that, and it’s pretty cool.

Step 6: Taking it Further

This is just the tip of Jekyll.


Well, that’s your introduction to Jekyll – the simple, blog aware, static site generator. The next time you’re building a brochure-style, business-card-y, micro-portfolio site, think you’ll give Jekyll a try? Let me know in the comments and thank you so much for reading!

Tags: Other jekyll Ruby

August 12 2011


How to Query Web Services with a Build Script

Do you hate that your favorite web service can’t be used in your build scripts? Well, are you sure that’s the case? I’ll show you how to use a popular tool, Phing – which is essentially a port of the popular Apache Ant for PHP – to query the Nettuts+ Prefixr API in today’s quick tip.

Choose 720p for the clearest picture.
Subscribe to our YouTube and channels to watch more screencasts.

Final Sample Phing Project

<?xml version="1.0"?>

<project name="demo" default="prefixr">
	<property name="css.path" value="css/style.css" />
	<target name="prefixr">
		<echo msg="Updating your stylesheet" />
		<loadfile property="css.min" file="${css.path}" />
		<exec command="curl -sSd css='${css.min}' > ${css.path}" />



The more I use Phing, the more I realize just how powerful it is. While many prefer Apache Ant for its portability, with Phing, I can use my existing PHP knowledge to create any new tasks or filters that I require. That’s incredibly powerful and convenient. What’s your build tool of choice?

July 17 2011


Build Communication Apps with Twilio: New Premium Tutorial

Advertise here

In this two-part Premium tutorial, we’re going to cover Twilio, a web service that allows you to integrate web applications with telecommunications. This is neat because you can use a phone to access your web app, or even your web app to access phones!

Become a Premium member to read this tutorial, as well as hundreds of other advanced tutorials and screencasts.

Build Communication Apps with Twilio: New Premium Tutorial

Join Net Premium

NETTUTS+ Screencasts and Bonus Tutorials

For those unfamiliar, the family of Tuts+ sites runs a premium membership service. For $9 per month, you gain access to exclusive premium tutorials, screencasts, and freebies from Nettuts+, Psdtuts+, Aetuts+, Audiotuts+, Vectortuts+, and CgTuts+ For the price of a pizza, you’ll learn from some of the best minds in the business. Become a Premium member to read this tutorial, as well as hundreds of other advanced tutorials and screencasts.

June 20 2011


How to Customize the Command Prompt

Advertise here

I’m a big fan of the terminal: whether you’re leveraging a handful of commands (or more!) to improve your development process, or just using it to quickly move around your drives and open files and folders, the command line is an awesome tool. However, if you use it often, you’ll want to customize it to your needs. I’ll show you how to do that today!

I’m often asked, “How did you get your command prompt to look like that?” Well, in this tutorial, I’ll show you exactly how to do it. It’s pretty simple, and won’t require too much of your time.

I should note that what I’m showing you is specifically for the bash shell; this is the default shell on Mac and most Linux systems. If you’d like a bash prompt on Windows, you might want to check out Cygwin.

How does it Work?

Before we get started, let’s talk for a minute about how you customize your bash prompt. It’s not quite the same as you’re average application: there’s no preferences panel. Your customizations are stored in a file. If you’re on Linux (or using Cygwin), that will be your .bashrc file; on Mac, that’s your .bash_profile file. In both cases, this file is kept in your home directory (if you aren’t sure where that is for a Cygwin install, run the command echo $HOME). Note that I’ll only refer to the .bashrc file from here on out, but use the .bash_profile if you’re on a Mac.

Note that on Macs (and possibly Linux machines; I’m not sure), files that begin with a period are hidden by default. To show them, run these two lines in the terminal

defaults write AppleShowAllFiles TRUE
killall Finder

So, what goes in that .bashrc file? Each line is actually a command that you could run on the command line. In fact, that’s how these config files work: When you open the console, all the commands you’ve written in the config file are run, setting up your environment. So, if you just want to try out some of what I’ll show below, just type it on the command line itself. The simplicity here is beautiful.

Customizing PS1

Let’s start with a definition. The prompt is what you see at the beginning of the line, each time you hit enter on the command line. Here’s what the default settings are for the Mac:

Mac Default Terminal

In this case, the prompt is andrews-macbook:~ screencast$. There’s a few variables here: andrew-macbook is the name of this computer, ~ is the current directory (the home directory) and screencast is the username. Let’s customize this a bit.

Open up either your .bashrc file. The way we set what information is displayed in the prompt is with the PS1 variable. Add this to the file:


Notice that I don’t put spaces on either side of the equals sign; that’s necessary. Save this file in your home directory and re-open a terminal window. Now, you should have a prompt that looks like this:

Mac Customized Terminal

I’ll note here that if you find it tedious to close and re-open your terminal each time you make a change to your .bashrc or .bash_profile, there’s a bit of a shortcut: You can load any bash customization file with the source command. Run this in your terminal:

source ~/.bashrc 

Still too long? Well, a single period (.) is an alias for source. Happy now? If you’re quick, you’ll realize that we can use the source command to include other files within our .bashrc file, if you want to split it up to keep in under control.

Let’s customize our prompt a bit more. We can use built-in variables in the string we assign to PS1 to include helpful information in the prompt; here’s a few useful one:

  • \d: Date
  • \h: Host
  • \n: Newline
  • \t: Time
  • \u: Username
  • \W: Current working directory
  • \w: Full path to current directory

So, if you set your prompt to this:


you should see something like this:

Mac Customized Terminal

Notice a few things here: firstly, we’re using a bunch of the variables shown above to give us more information. But secondly, we’re including a few newlines in there, and getting a more interesting prompt: we have the current directory on one line, and then the actual prompt on the next line. I prefer my prompt this way, because I always have the same amount of space to write my commands, no matter how long the path to the current directory is. However, there’s a better way to do this, so let’s look at that now.


The better way to do this is the use the PROMPT_COMMAND variable; the contents of this variable isn’t just a string, like with PS1. It’s actually a command that executed before bash displays the prompt. To give this a try, let’s add this to our .bashrc:

PROMPT_COMMAND='echo "comes before the prompt"'

We’re using the echo command here; if you aren’t familiar with it, you just pass it a string, and it will write it to the terminal. By itself, it’s not incredibly useful (although you can use it to view variables: echo $PS1), but it’s great when used with other commands, so display their output. If you added the line above, you should see this:

Mac Customized Terminal

Let’s do something more useful here. Let’s write a bash function that we will assign to PROMPT_COMMAND. Try this:

print_before_the_prompt () {
    echo "comes before the prompt"


If you use this, you shouldn’t see a difference in your prompt from what we have above. Now, let’s make this useful.

print_before_the_prompt () {
  echo "$USER: $PWD"



Here’s what you’ll get:

Mac Customized Terminal

That’s a good start, but I want to do a bit more. I’m going to use the printf command instead of echo because it makes including newlines and variables a bit easier. A quick background on the printf command: it takes several paramters, the first being a kind of template for the string that will be outputted. The other parameters are values that will be substituted into the template string where appropriate; we’ll see how this works.

So let’s do this:

print_before_the_prompt () {
    printf "\n%s: %s\n" "$USER" "$PWD"

See those %s parts in there? That means “interpret the value for this spot as a string”; for context, we could also use %d to format the value as a decimal number. As you can see, we have two %ss in the “template” string, and two other parameters. These will be placed into the string where the %ss are. Also, notice the newlines at the beginning and end: the first just gives the terminal some breathing room. The last one makes sure that the prompt (PS1) will be printed on the next line, and not on the same line as PROMPT_COMMAND.

You should get a terminal like this:

Mac Customized Terminal

Adding Some Color

Looking good! But let’s take it one step farther. Let’s add some color to this. We can use some special codes to change the color of the text in the terminal. It can be rather daunting to use the actual code, so I like to copy this list of variables for the color and add it at the top of my .bashrc file:

txtblk='\e[0;30m' # Black - Regular
txtred='\e[0;31m' # Red
txtgrn='\e[0;32m' # Green
txtylw='\e[0;33m' # Yellow
txtblu='\e[0;34m' # Blue
txtpur='\e[0;35m' # Purple
txtcyn='\e[0;36m' # Cyan
txtwht='\e[0;37m' # White

bldblk='\e[1;30m' # Black - Bold
bldred='\e[1;31m' # Red
bldgrn='\e[1;32m' # Green
bldylw='\e[1;33m' # Yellow
bldblu='\e[1;34m' # Blue
bldpur='\e[1;35m' # Purple
bldcyn='\e[1;36m' # Cyan
bldwht='\e[1;37m' # White

unkblk='\e[4;30m' # Black - Underline
undred='\e[4;31m' # Red
undgrn='\e[4;32m' # Green
undylw='\e[4;33m' # Yellow
undblu='\e[4;34m' # Blue
undpur='\e[4;35m' # Purple
undcyn='\e[4;36m' # Cyan
undwht='\e[4;37m' # White

bakblk='\e[40m'   # Black - Background
bakred='\e[41m'   # Red
badgrn='\e[42m'   # Green
bakylw='\e[43m'   # Yellow
bakblu='\e[44m'   # Blue
bakpur='\e[45m'   # Purple
bakcyn='\e[46m'   # Cyan
bakwht='\e[47m'   # White

txtrst='\e[0m'    # Text Reset

There’s some method to this madness: The first set are turn on normal coloring. The second set turn on bold coloring. The third set turn on underlined coloring. And that fourth set turn on background coloring. That last one resets the coloring to normal. So, let’s use these!

print_before_the_prompt () {
    printf "\n $txtred%s: $bldgrn%s \n$txtrst" "$USER" "$PWD"

Here, I’ve added $txtred before the first %s, and $bldgrn before the second %s; then, at the end, I’ve reset the text color. You have to do this because once you set a color, it will hold until you either use a new color or reset the coloring. You’ll also notice that when setting a variable, we don’t prefix it with a dollar sign; but we do use the dollar sign when using the variable: that’s the way bash variables work. This gives us the following:

Mac Customized Terminal

Let’s move on to the final step: adding some scripting to give us even more information.

Adding Version Control Information

If you’ve seen the screencasts that come with my book Getting Good with Git (yes, a shameless plug), you might remember that I have some version control information showing in my prompt. I got this idea from the excellent PeepCode “Advanced Command Line” screencast, which share this, as well as many other great tips.

To do this, we’re going to need to download and build the script that finds this information. Head on over to the repository for vcprompt, a script that outputs the version control information. If you’re familiar with the Mercurial version control system, you can use that to get the repo, but you’ll most likely want to hit that ‘zip’ link to download the script code as a zip file. Once you unzip it, you’ll have to build the script. To do this, just cd into the unzipped script folder and run the command make. Once this command runs, you should see a file named ‘vcprompt’ in the folder. This is the executable script.

So, how do we use this in our prompt? Well, this brings up an important rabbit-trail: how do we “install” a script (like this one) so that we can use it in the terminal? All the commands that you can run on the terminal are found in a defined array of folders; this array is the PATH variable. You can see a list of the folders currently in your PATH by running echo $PATH. It might look something like this:


What we need to do is put the executable script vcprompt in a folder that’s in our path. What I like to do (and yes, I learned this trick from that PeepCode screencast, too) is create a folder called ‘bin’ (short for ‘binary’) in my home directory and add that folder to my PATH. Add this to your .bashrc:

export PATH=~/bin:$PATH

This sets PATH to ~/bin, plus whatever was already in the PATH variable. If we now put that vcprompt script into ~/bin, we will be able to execute it in any folder on the terminal.

So, now let’s add this to our prompt:

print_before_the_prompt () {
    printf "\n $txtred%s: $bldgrn%s $txtpur%s\n$txtrst" "$USER" "$PWD" "$(vcprompt)"

I’ve added $txtpur %s to the “template” string, and added the fourth parameter"$(vcprompt)". Using the dollar sign and parenthesis will execute the script and return the output. Now, you’ll get this:

Mac Customized Terminal

Notice that if the folder doesn’t use some kind of version control, nothing shows. But, if we’re in a repository, we get the version control system that’s being used (Git, in my case) and the branch name. You can customize this output a bit, if you’d like: check the Readme file that you downloaded with the source code for the vcprompt script.

Moving On!

Here’s our complete .bashrc or .bash_profile file:

export PATH=~/bin:$PATH

txtblk='\e[0;30m' # Black - Regular
txtred='\e[0;31m' # Red
txtgrn='\e[0;32m' # Green
txtylw='\e[0;33m' # Yellow
txtblu='\e[0;34m' # Blue
txtpur='\e[0;35m' # Purple
txtcyn='\e[0;36m' # Cyan
txtwht='\e[0;37m' # White
bldblk='\e[1;30m' # Black - Bold
bldred='\e[1;31m' # Red
bldgrn='\e[1;32m' # Green
bldylw='\e[1;33m' # Yellow
bldblu='\e[1;34m' # Blue
bldpur='\e[1;35m' # Purple
bldcyn='\e[1;36m' # Cyan
bldwht='\e[1;37m' # White
unkblk='\e[4;30m' # Black - Underline
undred='\e[4;31m' # Red
undgrn='\e[4;32m' # Green
undylw='\e[4;33m' # Yellow
undblu='\e[4;34m' # Blue
undpur='\e[4;35m' # Purple
undcyn='\e[4;36m' # Cyan
undwht='\e[4;37m' # White
bakblk='\e[40m'   # Black - Background
bakred='\e[41m'   # Red
badgrn='\e[42m'   # Green
bakylw='\e[43m'   # Yellow
bakblu='\e[44m'   # Blue
bakpur='\e[45m'   # Purple
bakcyn='\e[46m'   # Cyan
bakwht='\e[47m'   # White
txtrst='\e[0m'    # Text Reset

print_before_the_prompt () {
    printf "\n $txtred%s: $bldgrn%s $txtpur%s\n$txtrst" "$USER" "$PWD" "$(vcprompt)"


Well, that’s a crash-course on customizing your bash prompt. If you have any questions, be sure to drop them in the comments!

June 16 2011


Python from Scratch: Variables, Data Types and Control Structure

Advertise here

Welcome back to Python from Scratch, where we’re learning Python…from scratch! In the last lesson, we installed Python and got set up. Today, we’re going to cover quite a bit, as we learn the essentials. We’ll review variables, operators, and then finish up by learning about control structures to manage the flow of your data.

Video Tutorial

Alternate Source
Subscribe to our YouTube and channels to watch more screencasts.


Variables are the first thing you should learn in any new language. You can think of them as named containers for any kind of data. The syntax to declare them is: name = value You can name anything you like (except for a handful of keywords), and their values can be any type of data.

Data Types

There are many data types, but the following four are the most important:


Numbers can be either integers or floating point numbers.

  • Integers are whole numbers
  • Floats have a decimal point


String are lines of text that can contain any characters. They can be declared with single or double quotes.

	empty = ""
	escaped = "Can\'t"
	greeting  = "Hello World"
	multiLine = "This is a long \n\
	string of text"

You have to escape single and double quotes within the string with a backslash. Otherwise, Python will assume that you’re using them to end the string. Insert line breaks with \n. Python also supports string interpolation using the percent symbol as follows:

name = "John Doe"
greeting = "My name is %s" % name

You can access sets of characters in strings with slices, which use the square bracket notation:

"Hello"[2] #outputs "l"


Booleans represent either a True or False value. It’s important to note that you have to make the first letter capital. They represent data that can only be one thing or the other. For example:

	isMale = True #Could be used in software with a database of users
	isAlive = False #Could be used in a game, set when the character dies


Lists are used to group other data. They are called Arrays in nearly all other languages. You can create a list with square brackets.

	emptyList = []
	numbersList = [1, 2, 3]
	stringsList = ["spam", "eggs"]
	mixedList = ["Hello", [1, 2, 3], False]

As you can see above, lists may contain any datatypes, including other lists or nothing at all.

You can access parts of lists just like strings with list indexes. The syntax is the same:

numbersList[1] #outputs 2
stringList[0] #outputs spam
mixedList[1][2] #outputs 3

If you nest a list within another list, you can access them with multiple indexes.


Comments are used to describe your code, in the case that you want to come back to it later, or work in a project with someone else.

#This a comment on it's own line
#You create them with the hash symbol
var = "Hello" #They can be on the same line as code


You’ve seen operators before. They’re those things like plus and minus, and you use them in the same way that you learned in school.

	2 + 3 #Addition, returns 5
	8 - 5 #Subtraction, returns 3
	2 * 6 #Multiplication, returns 12
	12 / 3 #Division, returns 4
	7 % 3 #Modulo, returns the remainder from a division, 1 in this case.
	3**2 #Raise to the power, returns 9

You can also assign the result of an operation on a variable back to the same variable by combining the operator with an equals sign. For example, a += b is a more concise version of a = a + b

	x = 2
	x += 4 #Adds 4 to x, it now equals 6
	x /= 2 #Divides x by 2, it now equals 3

Control Structures

Once you’ve created and manipulated variables, control structures allow you to control the flow of data. The two types we’re learning today are conditionals and loops.


Conditionals allow you to run different blocks of code based on the value of data.

a = 2
b = 3

if a < b:
    print "Success"


The two types of loops we’re discussing here are for loops and while loops. for loops work using lists, and while loops work using conditions.

while loops

a, b = 0, 5

while a < b:
	print a
	a += 1

for Loops

myList = [1, 2, 3, 4, 5]

for a in myList:
	print a


That's it for today, but we've covered a bunch of techniques. Feel free to run though everything a few times until it makes sense. I'll try and answer any more questions in the comments, and I hope you'll join me for the rest of the series!

May 05 2011


Python from Scratch: Getting Started

Advertise here

Welcome to Python from Scratch, where I’m going to teach you the ins and outs of Python development… from scratch.

In this first lesson, we’re going choose a version, install Python, and then create the obligatory “Hello world” script. If you’re already familiar with Python, feel free to skip ahead to a later lesson in the series.

Video Tutorial

Subscribe to our YouTube and channels to watch more screencasts.

Companion Article

Choosing a Version

“It’s important to choose the right version.”

There are two versions of Python that are currently being developed: 2.x and 3.x. It’s important to make the right choice ,because there are several differences between the two. Once you’ve learned one, it can be a bit annoying to have to transition to the other. In this series, we’ll be working through version 2.7.1. You may want to go this route in order to follow along with the videos and articles in this series. That said, most things should work with either version. Version two has much more support from third party libraries, whereas version three has more features, and plenty of bug fixes and refinements.

To make things easier, a lot of features that are being added to version three have also being added to version two, so there’s less need to worry about the differences.

Installing the Interpreter

Once you’ve chosen a version, it’s time to install. Download the version of Python for your OS, and run the installer which will get it set up on your machine. There are three ways you can now use Python:

  • Python Shell- lets you run commands line by line.
  • IDLE GUI – lets you write more complex scripts, and run them in one go.
  • Text Editor – any text editor that runs on you system. You can then save it with a .py extension, and run it from the shell.

For now, launch the shell to test if it works correctly. On Windows, navigate to the directory you installed Python. By default, it should be C:\Python27 Once there, launch python.exe. If you’re on Mac or Linux, launch the Terminal, then type python.

I personally find IDLE to be unpleasant to use; so for most of this series, we’re going to be using a standard code editor. If you would rather use an IDE, there are plenty of good ones:

Hello World!

No we’re all set up; let’s write your first bit of Python! Go to the shell, type print "Hello World!", and hit enter. The phrase Hello World! should appear.

And that’s it: it really is as simple as that. You’ve written your first Python program! Now is a good opportunity to highlight one of the differences beween version two and three: they changed print from a statement to a function. Don’t worry too much about what those words mean for now. All you need to know is that, if you chose to use version three, you need to write print("Hello World!") — the only difference is that you don’t use brackets. We’ll learn about the technical reasons behind this in a future lesson.


So in this lesson, we set ourselves up with a fresh Python installation, discussed the best tools to write code, and wrote our first program. If you have any questions, I’m happy to answer them in the comments, and I hope you’ll join me for the rest of this series.

May 02 2011


Vim Essential Plugin: PeepOpen

Advertise here

In this episode of our “Vim Essential Plugins” quick tip series, we’ll take a look at the only application in our list that isn’t free: PeepOpen. But, don’t let this deter you from picking up one of the most useful applications available for Vim users.

Subscribe to our YouTube and channels to watch more screencasts.



Like the other plugins that have been featured in this Session, PeepOpen is essentially a file explorer — but a beautiful and incredibly fast one at that. Somewhat uniquely, PeepOpen is an application that is stored in your menu bar.

PeepOpen Menu

You’ll find that this application isn’t limited only to Vim users, though. Nope – from Coda, to TextMate, to Emacs…all the best Mac editors are available. Sorry Windows users. You’re out in the cold for this one, unfortunately.


Upon viewing the preferences section of the app, you’ll find a helpful tab that allows you to specify which file types to ignore — such as DS_STORE, .swp, etc. The PeepOpen team have already taken care of the most pertinent file types for you.

Within MacVim, PeepOpen can be triggered by typing Control + o. At this point, you can type any sequence of characters to identify the file that you wish to access, and note that this is fuzzy searching.

And it’s really as simple as that. Rather than opening the file explorer, or wasting time searching through your buffers, simply use PeepOpen. It’ll save you an incredible amount of time over the course of the year.

April 21 2011


Quick Tip: The Power of Google Analytics Custom Variables

Familiarity with your users and their needs is the most important aspect of any successful site or web campaign. Google Analytics is the perfect tool to gather this sort of information. But there’s much more you can find out about your users when you begin using custom variables to make better decisions.

What are Custom Variables?

Using custom variables carries huge potential, as it allows you to acquire information about the behavior of your visitors.

You can treat custom variables as your own extension of metrics and dimensions. These provide you with the ability to gather non-standard and detailed data, which are not available via the Google Analytics panel.

Using custom variables carries huge potential, as it allows you to acquire information about the behavior of your visitors, which, in total, can significantly contribute to an increase of the ROI (return on investment) of the website or e-shop.

For example, using custom variables makes it possible to differentiate the activities of the logged in users versus those who never signed in. This provides an opportunity to observe the behavior of the website designed for a particular target group. For instance, we can learn which page of our website is the one that males – aged between 20 and 30 – visit most frequently. And this is just a tiny piece of the information that can be stored with the help of custom variables.

So How do Custom Variables Work?

The functionality of custom variables is wonderful in its simplicity. In fact, it’s based on the simple fact that, while performing a pre-defined activity, the user is labeled and the information about this label is subsequently stored in a cookie. Then, on the basis of this particular label, we can create a new section in the statistics of the Google Analytics panel.

Custom variables can be used in one of three ways:

  • page-level – this is the lowest level used to monitor particular ways in which the visitor interacts with the web page (e.g. AJAX, watching a video etc.)
  • session-level – a label attached to this level is added to the actions of the visitor throughout the session, and is deleted when the session cookie expires.
  • visitor-level – this is the highest level; the label is permanent and remains detached during subsequent visits until the visitor deletes the cookie or overwrites the value.

How Do I Configure Custom Variables?

Custom variables are quite easy to configure; you only need to add one line of code before the _trackPageview call.

_gaq.push(['._setCustomVar,INDEX, NAME, VALUE, OPT_SCOPE']);
  • INDEX (required) – determines a slot for a custom variable in Google Analytics. There are 5 slots available, numbered 1 to 5. You should remember that, if the variables are to function correctly, a single variable must be placed in a single slot.
  • NAME (required) – is the name of the variable that is going to appear in the report of the Google Analytics panel – Custom Variables
  • VALUE (optional) – is the actual value of the variable which is paired with NAME. Any number of such value pairs is possible, for instance if NAME=country, VALUE can, one by one, equal the values of US, GB, PL etc.
  • OPT_SCOPE (optional) – is a level within which a custom variable functions. As I described above, there are three levels: 1 (visitor-level), 2 (session-level), 3 (page-level). When this parameter is not provided, it is replaced with the default page-level.
        var _gaq = _gaq || [];
        _gaq.push(['._setCustomVar,INDEX, NAME, VALUE, OPT_SCOPE']);

Some Practice

Now, let’s review how custom variables function in practice. Let’s suppose that we want to keep track of the visitors of our website’s – distinguishing between those who did and did not log in. In order to do so, before the _trackPageview call, we insert code describing the user.

			1, 			    // first slot
			'user-type',    // custom variable name
			'visitor', 	    // custom variable value
			2			    // custom variable scope - session-level

Once the visitor logs into your website, we change this code, accordingly:

			1, 			    // first slot
			'user-type',    // custom variable name
			'regular-user', // custom variable value
			2			    // custom variable scope - session-level

But What Follows?

It’s time to present the results of the described script. After the script had been executing for a week, an advanced segment in Google Analytics panel was created. Its aim is to divide the data in the panel into: total, report for logged-in users, and report for users who didn’t log in for particular metrics.

The segment itself is created through Advanced Segments => Create a new advanced segment. Then you should set the dimensions according to the screenshot below:

The variable which we defined using JavaScript was in the first slot, so we have to select Key 1 and Value 1. Then, we set the key which we are interested in (user-type), and the value for the defined key (visitor) together using concatenation. Next, we name and test the advanced segment. The number of visits during a particular period of time for particular segments will be calculated within the test.

We define the second segment which takes into account the logged-in users in the same way. We can create it by analogy with the pattern presented above – with the difference in that the custom variable is set as regular-user.

After establishing the two segments, we can activate them. The result is presented below. Such a set of data is a great basis for an in-depth analysis of the activities on a webpage.

It’s Worth Remembering…

  • Don’t duplicate the names of custom variables between the slots.
  • Set custom variables before the pageview call.
  • You cannot use more than five custom variables in a single request.
  • Consider event tracking instead of custom variables for some demands, as then no additional (false) pageviews are generated in the panel.
  • You can determine whether custom variables work by observing requests within Firebug, or with the help of the Chrome extension, Google Analytics Tracking Code Debugger.

This is Only the Beginning

Using custom variables translates into more justified and accurate site decisions.

The example presented in this article only illustrate the process of using a single custom variable, and determining the best way to manage a website, according to the type of visitor. Of course, this is only the beginning. Custom variables can become incredibly powerful when we combine several of them at the same time. As a quick example, with the applicable data from a website’s registration process, we can then track, not only the sex of the visitor (session-level), but also his or her age bracket (session-level). Further, we could divide the visitors into groups who have made purchases in our fictional eShop, or even track those who took a specific action, such as clicking on a Facebook button.

These techniques translate into more justified and accurate site decisions.


Vim Essential Plugin: NERDTree

In this episode of Vim Essential Plugins, we’ll review the fantastic NERDTree plugin, which is a much improved replacement for the traditional file explorer.

Subscribe to our YouTube and channels to watch more screencasts.


Begin by downloading the plugin to your Desktop (or any directory, really), and installing it.

cd ~/Desktop
git clone
cd nerdtree

With those few lines of code, the plugin is now installed! To open a NERDTree panel, in normal mode, we call :NERDTree.


At this point, we can open any file by typing o, or with the more convenient alias, the Enter key. Unlike the default file browser, this will open the new file directly into the buffer to the right of NERDTree, similar to what you might be used to in a program, like TextMate.

If you ever forget what the correct key is for a particular action, press ? to display a quickie help buffer.


To expedite the process of navigating through your directory structures, NERDTree allows you to conveniently create bookmarks. Do so by moving the cursor to the directory that you wish to bookmark, and then typing:

:bookmark <desired bookmark name>

With this in place, you can now bring up your list of available bookmarks by pressing B. It’s a huge help!

Bookmark Section


Press the letter m to bring up a menu that will allow you to quickly add, move, copy, and delete nodes (or files). So, for example, if I want to create a new html file within a particular directory, I can type ma newfile.html.

The menu bar

There are plenty more useful shortcuts available, but I’ll leave it to you to discover them. Hint – research the cd and C commands; I use them religiously. Additionally, refer to the screencast above for more shortcuts.

April 18 2011


The Perfect Workflow, with Git, GitHub, and SSH

In this lesson, we’ll focus on workflow. More specifically, we’ll use the helpful GitHub service hooks to automatically update a project on our personal server whenever we push updates to a GitHub repo.

Prefer a Video Tutorial?

Press the HD for a clearer picture.
Subscribe to our YouTube and channels to watch more screencasts.

Step 1 - Create a Git Repo

We certainly need some sort of project to play around with, right? Let’s do that right now. Using which ever tool you prefer (I’d recommend Structurer), create a new directory, called awesomeProject, and add an index.html file. Feel free to populate this with some gibberish markup for the time being.

With our test directory in place, let’s create our first Git commit.

If you’re unfamiliar with Git, I highly recommend that you first review “Easy Version Control with Git.”

Open the command line:

cd path/to/awesomeProject
git init
git add .
git commit -m 'First commit'

Those familiar with Git should feel right at home. We’re creating a Git repo, adding all the files to the staging area, and are then creating our first commit.

Step 2 - Uploading to GitHub

The next step is to upload our project to GitHub. That way, we can easily call a git pull to download this project from any computer/server we wish.

Again, if you’re not familiar with GitHub, and haven’t yet created an account, read Terminal, Git, and GitHub for the Rest of Us.

Begin by creating a new Git repository.

Create a Git Repo

Next, you’ll need to fill in some details about your project. That’s simple:


And finally, since we’re already working with an existing Git repo, we only need to run:

git remote add origin
git push -u origin master

With that out of the way, our awesomeProject is now available on GitHub. That was easy!

Step 3 - SSH

Now, we certainly need some sort of live preview for our project, ideally stored on our own server. But this can be a pain sometimes. Push your updates to GitHub, login to your server, manually transfer the updated directory, etc. Granted, this only takes a moment or so, but when you make multiple changes through out the day, this can quickly become a burden.

But one step at a time. We’ll tackle this dilemma in Step 4. For now, let’s simply pull in our Git repo to our server. To do so, we need to SSH in.

Depending upon your host, your SSH credentials will vary slightly. Search Google for “your-host-name SSH,” and you’ll surely find the necessary instructions. Once you’re ready, let’s move along:

We’ll use my personal server as an example:

<enter your host password>

And with those two lines, we’re in!


Next, we cd to the parent directory of where we wish to store awesomeProject. For me, this will be: cd domains/ Of course, modify this according to your own directory structure.

Git Clone

Let’s clone the GitHub repo now.

git clone

Give that command a few seconds, but, before you know it, that directory is now available on your server, and, in my case, could be viewed at:

Step 4 - Creating a Connection

The inherent problem at this point is that there’s no specific connection between our GitHub repo and the directory stored on our server — at least not an automated connection. For example, if we update our source files on our local machine, and then push the changes to GitHub:

git add index.html
git commit -m 'Added photo of dancing chicken'
git push origin master

These changes will certainly not be reflected on our server. Of course they won’t! In order to do so, we must – once again – SSH into our server, cd to the awesomeProject directory, and perform another git pull to bring in the updated source files.

Wouldn’t it be great if, every time we pushed updates to GitHub, those new source files were automatically updated on our live preview server?

As it turns out, we can do this quite easily with GitHub service hooks.

GitHub Service Hooks

You can access this page by pressing the “Admin” button from within your GitHub repo, and then clicking “Service Hooks.” The “Post-Receive URL” option will instruct GitHub to send a POST request to the specified page every time you push to your GitHub repo. This is exactly what we need!

“We’ll hit these URLs with POST requests when you push to us, passing along information about the push.”

To make this work, we’ll need to create one more file that will handle the process of performing the git pull. Add a new file, called github.php (or anything you wish – preferably more vague), and add:

<?php `git pull`;

So now you’re thinking: “Jeff’s gone crazy. You can’t put a Bash script into a PHP string.” Well…yes you can, once you realize that those aren’t single quotes above, they’re back-ticks.

When you wrap a sequence in back-ticks, in PHP, it’ll be treated as a Bash script. In fact, it’s identical to using the bash_exec function.

Save that file, and upload it to the awesomeProject directory on your server. When finished, copy the url to that file, and paste it into the “Post-Receive URL” textbox. In my case, the url would be

With this in place, every single time you push to your GitHub repo, that file will be called, and the awesomeProject directory on your server will auto-update, without you needing to move a finger. Pretty nifty, ay?

You Also Might Enjoy:

April 05 2011


Vim Essential Plugin: Markdown to HTML

We’re cheating a bit in this episode. Rather than installing an actual plugin, we’ll be using a couple tools to allow us to write and convert Markdown to HTML. It’s really a simple process, and can save great deals of time.


First, we need to download Markdown and place it within our runtime folder. On the Mac, you can drag the file into your usr/local/bin/ folder. And that’s it; you’re all set! Next, to trigger the markdown-to-html conversion, we need to provide two parameters: the path to the file, and the command itself.

Within Vim, the following should do the trick:

:%! /usr/local/bin/ --html4tags

This instructs the file to run on the curent page. But, that’s a little too much typing; let’s instead save this to our .vimrc file, and assign it to a shortcut.

".vimrc file

"Markdown to HTML
nmap md :%!/usr/local/bin/ --html4tags 

With this line of code, whenever we type – by default – backslash + md, that command will run, thus converting our markdown to clean HTML.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...