August 19, 2013

Last time we talked about getting ourselves set up to develop a web application in Flask. This week I'll expand on the concepts developed in Part I to create an application with the help from some Flask Extensions. We'll explore some more best practices for writing views, templates, models, integrating and managing a database, static files, and forms.

Today we're going to be building a basic web tracking/analytics application. The application will function as a platform for users to track who visited their website and what features on that website were used.

In terms of functionality-

  • Users sign up/log in and out,
  • Users specify which Sites they'd like to track,
  • Sites record Visits, and
  • Users view each Site's Visits.

Based on what we looked at last week our application should have a Users module and a Tracking module (which contains both Sites and Visits).

You can grab the repo here to get the code from Part 1. We'll be starting with that basic structure for Part 2. Please note: I made a mistake in Part 1: We need to include an file in each directory we want to use as a module. The final structure for the app in Part 1 looked like this:

├── app
│   ├──
│   ├──
│   ├── module
│   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   └──
│   ├── static
│   └── templates
├── app.db
├── docs
│   ├── config.html
│   ├── pycco.css
│   ├── run.html
│   └── shell.html
├── requirements.txt

Before we start, let's setup the main folder structure for Part 2. And before that, create a virtualenv within part_2:

$ virtualenv --no-site-packages myenv
$ source myenv/bin/activate

Then install the dependencies from the requirements.txt file:

$ pip install -r requirements.txt 

Now create the folder structure using the app from Part 1:

  1. Rename the module folder to users
  2. Copy the entire users folder and paste it in the same directory, then rename it to tracking
  3. Create a forms, tracking, and users directory in the templates directory

Current structure:

part 2
├── app
│   ├──
│   ├──
│   ├── static
│   ├── templates
│   │   ├── forms
│   │   ├── tracking
│   │   └── users
│   ├── tracking
│   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   └──
│   └── users
│       ├──
│       ├──
│       ├──
│       ├──
│       ├──
│       └──
├── app.db
├── docs
│   ├── config.html
│   ├── pycco.css
│   ├── run.html
│   └── shell.html
├── requirements.txt

Note: You can see the completed structure of Part 2 at the bottom of the post.


Let's start with the Users module. Users need to be able to sign up for an account, log in and log out. Ideally there would/could be more user functionality (edit profile, delete account, etc.) but for this application we'll stick with the basics.


In order to add functionality quickly, we'll use these Extensions:

  1. Flask-Login to help us manage user log ins and sessions.
  2. Flask-SQLAlchemy combined with psycopg2 as an ORM for interacting with the database.
  3. Flask-Heroku to help manage our settings and configuration because we'll be deploying on Heroku in a latter lesson.
  4. Flask-WTForms to help generate and validate HTML forms.

Go ahead and install these now using pip:

$ pip install flask-login==0.2.6 flask-sqlalchemy==1.0 psycopg2==2.5.1 flask-heroku==0.1.4 flask-wtf==0.8.4 wtforms==1.0.4

First, lets get Flask-Login, Flask-Heroku and Flask-SQLALchemy set up. According to Flask-Login's documentation all we have to do is add a few lines to our application initialization (app/ code and we'll be up and running. This follows for the other two Extensions as well:

from flask import Flask, render_template, send_from_directory
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.heroku import Heroku
from flask.ext.login import LoginManager
import os

app = Flask(__name__) #create our application object
app.config.from_object('config') #load our local config file

heroku = Heroku(app) #create a heroku config object from our app object

login_manager = LoginManager(app) #create a LoginManager Object from our app object

db = SQLAlchemy(app) #create a db (SQLAlchemy) object from our app object

#register the users module blueprint
from app.users.views import mod as usersModule

#add our view as the login view to finish configuring the LoginManager
login_manager.login_view = "users.login_view"

#register the tracking module blueprint
from app.tracking.views import mod as trackingModule

# controllers

def favicon():
    return send_from_directory(os.path.join(app.root_path, 'static'), 'ico/favicon.ico')

def page_not_found(e):
    return render_template('404.html'), 404

def index():
    return render_template('index.html')

Models and Views

Next lets take a look at our view and controller functions for Users to start the log in/log out functionality. In users/ we have:

from flask import Blueprint, render_template, flash, redirect, session, url_for, request, g
from flask.ext.login import login_user, logout_user, current_user, login_required
from app import app, db, login_manager
from forms import LoginForm, RegistrationForm
from app.users.models import User

mod = Blueprint('users', __name__) #register the users blueprint module

def load_user(user_id):
    return User.query.get(user_id)

@mod.route('/login/', methods=('GET', 'POST'))
def login_view():
    form = LoginForm(request.form)
    if form.validate_on_submit():
        user = form.get_user()
        flash("Logged in successfully.")
        return redirect(request.args.get("next") or url_for("index"))
    return render_template('users/login.html', form=form)

@mod.route('/register/', methods=('GET', 'POST'))
def register_view():
    form = RegistrationForm(request.form)
    if form.validate_on_submit():
        user = User()
        return redirect(url_for('index'))
    return render_template('users/register.html', form=form)

def logout_view():
    return redirect(url_for('index'))

Notice the load_user function. Flask-Login requires us to define this function in order to utilize its functionality. From this we can define the main three views for our Users module: login_view, register_view, and logout. Each of the three bits of functionality we defined for Users ended up as a view. This is just one way to do it - a view can handle more than one bit of functionality. In other words, you could put all the views for the entire application in one file - but that can get confusing. This level of separation makes it clear what bit of code manages what functionality.

Before we dive into what the code inside of those views does, let's define our User model:

from app import db
from app.mixins import CRUDMixin
from flask.ext.login import UserMixin
from app.tracking.models import Site

class User(UserMixin, CRUDMixin,  db.Model):
    __tablename__ = 'users_user'
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(50), unique=True)
    email = db.Column(db.String(120), unique=True)
    password = db.Column(db.String(120))
    sites = db.relationship('Site', backref='site',

    def __init__(self, name=None, email=None, password=None): = name = email
        self.password = password

    def __repr__(self):
        return '<User %r>' % (

We're only defining one model in here, because our Users module only has one model/object. Next notice that the User class subclasses three different objects.

The first is UserMixin, which comes from Flask-Login.

To make implementing a user class easier, you can inherit from UserMixin, which provides default implementations for all of these methods. (It’s not required, though.)

This fills in the other class functions our User object needs to implement in order for Flask-Login to work properly.

Next is a custom mixin which we borrowed from Flask-Kit. Create a new file called /app/ and add the following code:

from app import db

class CRUDMixin(object):
    __table_args__ = {'extend_existing': True}

id = db.Column(db.Integer, primary_key=True)

def get_by_id(cls, id):
    if any(
        (isinstance(id, basestring) and id.isdigit(),
         isinstance(id, (int, float))),
        return cls.query.get(int(id))
    return None

def create(cls, **kwargs):
    instance = cls(**kwargs)

def update(self, commit=True, **kwargs):
    for attr, value in kwargs.iteritems():
        setattr(self, attr, value)
    return commit and or self

def save(self, commit=True):
    if commit:
    return self

def delete(self, commit=True):
    return commit and db.session.commit()

When I built this application for the first time I noticed that I was re-implementing a lot of the same functionality across all the models. Instead of writing the same (or similar) code three times and then having to maintain it in all three spots it makes sense to break it out into its own class that the models can inherit from. This way if I need to change something in the future I only have to change it in one place.

CRUDMixin defines:

get_by_id(cls, id)

def create(cls, **kwargs)

def update(self, commit=True, **kwargs)

def save(self, commit=True)

def delete(self, commit=True)

Which is the start of a CRUD interface for the models.

Finally User subclasses db.Model comes from SQLAlchemy. This is our ORM. By subclassing this we have an automated way of relating this python class with a database table. This will manage all of our model creations, updates and deletions. Instead of handwriting SQL in your views or anywhere in your project this will talk to the database for you.

Next we define our User's attributes. Each user needs a database id, a name, an email and a password. Additionally we're making a has many relationship between Users and Sites.

Now going back to the view code, in register we need to make a new user:

@mod.route('/register/', methods=('GET', 'POST'))
def register_view():
    form = RegistrationForm(request.form)
    if form.validate_on_submit():
        user = User()
        return redirect(url_for('index'))
    return render_template('users/register.html', form=form)

Before we figure that out lets analyze the anatomy of a view.

The view starts with a decorator that specifies which url route and methods correspond with this view (in Django we'd specify these in Each view has access to a request object and any parameters that it is passed.

We can use Flask-WTF to define RegistrationForm and LoginForm in users/

from import Form, TextField, PasswordField, BooleanField, RecaptchaField, fields, validators
from import Required, Email, EqualTo
from app.users.models import User
from app import db

class LoginForm(Form):
    name = fields.TextField(validators=[validators.required()])
    password = fields.PasswordField(validators=[validators.required()])

    def validate_login(self, field):
        user = self.get_user()

        if user is None:
            raise validators.ValidationError('Invalid user')

        if user.password !=
            raise validators.ValidationError('Invalid password')

    def get_user(self):
        print, "here"
        return db.session.query(User).filter_by(

class RegistrationForm(Form):
    name = fields.TextField(validators=[validators.required()])
    email = fields.TextField(validators=[validators.Email()])
    password = fields.PasswordField(validators=[validators.required()])
    conf_password = fields.PasswordField(validators=[validators.required()])
    def validate_login(self, field):
        if db.session.query(User).filter_by( > 0:
            raise validators.ValidationError('Duplicate username')

These two forms handle passing the data from the template to the view. Inside of register_view we first check if the form is valid and complete, if it is we attempt to create a new user with the data. If this is successful we then save the user to the database. login_view works in a similar way but instead of creating a user it retrieves one from the database.


Flask comes with Jinja built in.

Jinja2 is one of the most used template engines for python. It is inspired by Django's templating system but extends it with an expressive language that gives template authors a more powerful set of tools. On top of that it adds sandboxed execution and optional automatic escaping for applications where security is important.

For each view we need to build an accompanying template. Each template inherits from templates/base.html, which we'll create now.

Note: To save space, I will provide the URL for the templates. Pull the code from there.

Grab the code here for base.html.

Notice the {% block _____ %}{% endblock %} statements. When we inherit from base.html we can define blocks in those templates and fill them in. The result will be a base.html shell with individual fillings. This allows us to reuse code and frees us from having to change some html code in a million different places because our product name changed or one of our nav links was axed.

In many cases, it's good practice to have a content block, a navigation block and a footer block. But again, it really depends on the application. There is a lot you can do with Flask and Jinja. So I suggest looking through the docs and at other people's projects to see whats possible.

Our template for registration_view is located at app/templates/users/register.html. It's good practice to have your module's template directory mirror your module's directory/view structure. Download here.

Our register.html is simple. This is because we are extending base.html and because we can build a template macro - templates/forms/macros.html - to make it easier to incorporate our forms in a uniform way.

Download the macro.

Instead of writing the same form html over and over again we can just use render_field to automatically generate the boilerplate html for each field in our forms. This way all of our forms will look and function in the same way and if we ever have to change them we only have to do it in once place. Are you starting to see a pattern here? Don't Repeat Yourself.

When Flask goes to serve up our registration page it first loads up the registration template, sees that it extends base.html, pulls that in and then keeps chugging until the html is all ready and compiled to ship over to the client.

Now add the login.html file, which follows the same logic as register.html:

Our application now has support for Users!

Finally, let's add the remaining templates (templates/404.html and templates/index.html), both of which can be found here.

Before moving on, I suggest going back over the first part of this to ensure that you understand what's happening. We went through this fast. Take the time to go over to the code. Learn it.

Tracking Requests

Next let's work on the meat of the application: the request tracking.

Request tracking works in a simple way. We generate a snippet of javascript, in this case a jQuery plugin, that you embed on the page you want tracked. When someone loads that page it sends a request to our application, which we store. We now know that someone visited, what ip address (and perhaps the location) they visited from, when they visited and what they did while they visited. This is powerful information. We can use this information to figure out what features to focus on, which features aren't being used, and where most of are users are from, etc.

I won't spend a lot of time on visualizing the data. Instead we'll focus on adding another module to our application with some custom code.

Our tracking applet uses two models: Sites and Visits. A User has many Sites, a Site belongs to one User. A Visit belongs to a Site <=> or a Site has many Visits. This is how we think about our models relating to each other. Speaking of models, the following is the code for in our tracking app:

from app import db
from app.mixins import CRUDMixin

class Site(CRUDMixin, db.Model):
    __tablename__ = 'tracking_site'
    id = db.Column(db.Integer, primary_key=True)
    visits = db.relationship('Visit', backref='tracking_site',
    base_url = db.Column(db.Text)
    user_id = db.Column(db.Integer, db.ForeignKey(''))

    def __init__(self, user_id=None, base_url=None):
        self.user_id = user_id
        self.base_url = base_url

    def __repr__(self):
        return '<Site %r>' % (self.base_url)

class Visit(CRUDMixin, db.Model):
    __tablename__ = 'tracking_visit'
    id = db.Column(db.Integer, primary_key=True)
    browser = db.Column(db.Text)
    date = db.Column(db.DateTime)
    event = db.Column(db.Text)
    url = db.Column(db.Text)
    site_id = db.Column(db.Integer, db.ForeignKey(''))
    ip_address = db.Column(db.Text)
    location = db.Column(db.Text)
    location_full = db.Column(db.Text)

    def __init__(self, browser=None, date=None, event=None, url=None, ip_address=None, location_full=None, location=None):
        self.browser = browser = date
        self.event = event
        self.url = url
        self.ip_address = ip_address
        self.location_full = location_full

    def __repr__(self):
        return '<Visit %r - %r>' % (self.url,

You can see the foreign keys, which relate specific fields from different tables.

A User registers a new site and then embeds that Site's tracking code on that site. That Site then sends requests that get logged as Visits to that Site.

We'll use two different views to account for the required functionality:

from flask import Blueprint, Response, render_template, flash, redirect, session, url_for, request, g
from flask.ext.login import current_user, login_required
from app import app, db, login_manager
from app.tracking.models import Site, Visit
from app.tracking.forms import RegisterSiteForm
from datetime import datetime
from app.tracking.geodata import get_geodata
from app.tracking.decorators import crossdomain

mod = Blueprint('tracking', __name__)

@mod.route('/sites/', methods=('GET', 'POST'))
def sites_view():
    form = RegisterSiteForm(request.form)
    sites = current_user.sites.all()
    if form.validate_on_submit():
        site = Site()
        site.user_id =
        return redirect('/sites/')
    return render_template('tracking/index.html', form=form, sites=sites)

#<%= %>/visited?event='+tracker.settings.event+'&data=''&visitor='+tracker.settings.visitor
@mod.route('/visit/<int:site_id>/visited', methods=('GET','POST'))
@crossdomain(origin="*", methods=["POST", "GET, OPTIONS"], headers="Content-Type, Origin, Referer, User-Agent", max_age="3600") 
def register_visit(site_id):
    site = Site.get_by_id(site_id)
    if site:
        browser = request.headers.get('User-Agent')
        date =
        event = request.args.get('event')
        url = request.url
        ip_address = request.remote_addr
        geo = get_geodata(ip_address)
        location_full = ", ".join([geo['city'],geo['zipcode'],geo['latitude'],geo['longitude']])
        location = ", ".join([geo['city'],geo['zipcode']])
        visit = Visit(browser, date, event, url, ip_address, location_full, location)
        visit.site_id = site_id
    return Response("visit recorded", content_type="text/plain")

# self, browser=None, date=None, event=None, url=None, ip_address=None, location_full=None
  • sites_view which will allow users to register sites and view the visits to sites in one spot, and
  • register_visit which is what will record the hits from our sites

I've elected to use sites_view for both displaying sites and their visits and for registering sites to demonstrate that a view can perform more than just one bit of functionality. It's really up to you how you want to break up your application. I've decided that instead of having a user go to a separate page that's sole function is to register sites, they can just do that in the same place the information is displayed.

In a production environment, you'd probably want to break all of these apart into a RESTful interface for all of the models. Instead of having Python/Flask/Jinja serve up a pre-formatted page you would likely use a javascript MVC framework to handle the front-end and make requests to the backend to fetch the necessary data. The client would then send requests to the server to make/register new sites and be in charge of updating the views when new sites/visits are created. The views would then be responsible for the REST interface.

That said, since we are focusing on Flask, we will use Jinga to serve up the page.

Take a look at the sites_view.

The @login_required decorator is provided by Flask-Login. Anyone who isn't logged in who tries to go to /sites/ will be redirected to the login page. This script supports both GET and POST requests at /sites/. This is because this view both has to serve the form and handle its posts. Alternatively, we could handle the posts in a separate view.

register_visit, meanwhile, is a bit more complex. The request tracking work likes this:

A User embeds the jQuery script in one of their websites. When a visitor loads up that website the jQuery script sends a request to our site. Because that request comes from another website its considered a "cross domain request". Generally this is a security issue. Read more about the Same Origin Policy here.

Lucky for us when I googled "cross origin requests flask" I found this. Armin Ronacher has written a nice little decorator that allows us to specify where we allow our requests to come from:

from datetime import timedelta
from flask import make_response, request, current_app
from functools import update_wrapper

def crossdomain(origin=None, methods=None, headers=None,
                max_age=21600, attach_to_all=True,
    if methods is not None:
        methods = ', '.join(sorted(x.upper() for x in methods))
    if headers is not None and not isinstance(headers, basestring):
        headers = ', '.join(x.upper() for x in headers)
    if not isinstance(origin, basestring):
        origin = ', '.join(origin)
    if isinstance(max_age, timedelta):
        max_age = max_age.total_seconds()

    def get_methods():
        if methods is not None:
            return methods

        options_resp = current_app.make_default_options_response()
        return options_resp.headers['allow']

    def decorator(f):
        def wrapped_function(*args, **kwargs):
            if automatic_options and request.method == 'OPTIONS':
                resp = current_app.make_default_options_response()
                resp = make_response(f(*args, **kwargs))
            if not attach_to_all and request.method != 'OPTIONS':
                return resp

            h = resp.headers

            h['Access-Control-Allow-Origin'] = origin
            h['Access-Control-Allow-Methods'] = get_methods()
            h['Access-Control-Max-Age'] = str(max_age)
            if headers is not None:
                h['Access-Control-Allow-Headers'] = headers
            return resp

        f.provide_automatic_options = False
        return update_wrapper(wrapped_function, f)
    return decorator

Stick the above code in /tracking/ (if you needed to use it in more than one module, then you'd put it in a global decorator file).

Take a look at the route in the register_visit function. In the route we've stuck a parameter site_id. When someone hits, for example, we pass the id of the site into the view. This allows us to recognize which site the request corresponds to.

This isn't a very solid implementation of this idea. Someone could modify the jQuery code or submit modified requests and we wouldn't know which site it actually came from. But this is simple and it easy to implement. But do not use this code in a production environment.

Next add a form:

from import Form, TextField, PasswordField, BooleanField, RecaptchaField, fields, validators
from import Required, Email, EqualTo
from app.users.models import User
from app import db

class RegisterSiteForm(Form):
    base_url = fields.TextField(validators=[validators.required()])

get_geodata(ip_address) calls a script that queries this is so we can get a rough idea of where the requests are coming from:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

Searches Geolocation of IP addresses using
It will fetch a csv and return a python dictionary

sample usage:
>>> from freegeoip import get_geodata
>>> get_geodata("")

{'status': True, 'city': 'Niter\xc3\xb3i', 'countrycode': 'BR', 'ip': '', 
'zipcode': '', 'longitude': '-43.0944', 'countryname': 'Brazil', 'regioncode': '21', 
'latitude': '-22.8844', 'regionname': 'Rio de Janeiro'}

from urllib import urlopen
from csv import reader
import sys
import re

__author__="Victor Fontes Costa"
__copyright__ = "Copyright (c) 2010, Victor Fontes -"
__license__ = "GPL"
__version__ = "2.1"
__maintainer__ = __author__
__email__ = "contato [a]"
__status__ = "Development"


def valid_ip(ip):

    pattern = r"\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"

    return re.match(pattern, ip)

def __get_geodata_csv(ip):
    if not valid_ip(ip):
        raise Exception('Invalid IP format', 'You must enter a valid ip format: X.X.X.X')

    response_csv = reader(urlopen(URL))
    csv_data =

    return {
        "status": u"True" == csv_data[0],

def get_geodata(ip):
    return __get_geodata_csv(ip)

if __name__ == "__main__":     #code to execute if called from command-line
    intput_ip = sys.argv[1]
    geodata = get_geodata(intput_ip)
    print "IP: %s" % geodata["ip"]
    print "Country Code: %s" % geodata["countrycode"]
    print "Country Name: %s" % geodata["countryname"]
    print "Region Code: %s" % geodata["regioncode"]
    print "Region Name: %s" % geodata["regionname"]
    print "City: %s" % geodata["city"]
    print "Zip Code: %s" % geodata["zipcode"]
    print "Latitude: %s" % geodata["latitude"]
    print "Longitude: %s" % geodata["longitude"] 

Save this as in the tracking directory.

Return to the view, all this view is doing is copying info from the request down and storing it in the database. It responds to the request with a simple string: visit recorded. The jQuery script on the other end doesn't really care about the response.

Finally, let's add in the template. Save this as index.html within the templates/tracking directory.

Remaining Files

We also need to setup a configuration file,

import os
_basedir = os.path.abspath(os.path.dirname(__file__))

DEBUG = False

ADMINS = frozenset([''])
SECRET_KEY = 'SecretKeyForSessionSigning'


CSRF_SESSION_KEY = "somethingimpossibletoguess"

RECAPTCHA_PUBLIC_KEY = 'blahblahblahblahblahblahblahblahblah'
RECAPTCHA_PRIVATE_KEY = 'blahblahblahblahblahblahprivate'
RECAPTCHA_OPTIONS = {'theme': 'white'}

You can read more about this file here.

The starts the server:

from app import app is used for running an interactive shell:

#!/usr/bin/env python
import os
import readline
from pprint import pprint

from flask import *
from app import *

os.environ['PYTHONINSPECT'] = 'True'

You can find more information on this file here.

Getting The DB set up

Because we will end up deploying this app to Heroku in a latter tutorial, which has Postgres support, we don't really need to run a database locally. But let's go ahead and test it out anyway for practice. It's easy to do. Flask-SQLAlchemy provides some nice functions to get your database configured for you.

First you want to make sure you've got a database you can store things in. Let's go with SQLite since we don't have to mess with database configuration. To get our app working with SQLite, add the following to the file:

DATABASE = 'test.db'
DATABASE_PATH = os.path.join(_basedir, DATABASE)

Now you can generate the tables.

$ python
>>> from app import db
>>> db.create_all()
>>> exit()

Unless you have errors this will generate all of your tables for you. If you have to change something you'll have to use what's called a database migration (which we'll cover next time).

Running the App

With your database setup, go ahead and run your app:

$ python

Wrapping Up

This week we put what we learned last time to work. We made a functioning web tracker and user framework.

Some of the main design ideas were to not repeat code (DRY) and break up our ideas down by functionality and then implement them.

You can check out a working copy of the app here. Sign up and register some sites, embed the jQuery and then see it rack up visits. The code for the app can be found here. But hopefully you already have it, as you followed the tutorial. :)

Next time I'll cover writing tests for your application, debugging errors, and database migrations.

Your application directory should look like this once complete:

├── app
│   ├──
│   ├──
│   ├──
│   ├── users
|   |   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   └──
│   ├── tracking
|   |   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   ├──
│   │   └──
│   ├── static
│   └── templates
|   |   ├── forms
|   |   |   └── macros.html
|   |   ├── tracking
|   |   |   └── index.html
|   |   ├── users
|   |   |   ├── login.html
|   |   |   └── register.html
|   |   ├── 404.html
|   |   ├── base.html
|   |   └── index.html
├── app.db
├── docs
│   ├── config.html
│   ├── pycco.css
│   ├── run.html
│   └── shell.html
├── requirements.txt
├── Procfile

blog comments powered by Disqus