It is usually a bad idea to measure programmer’s efficiency in numbers. There is a lot of useless metrics and each one brings negative (even disruptive) side effects. Once I got a piece of code that had been written by someone who literally was been paid for each line of code. That was a module of about 9000 lines of JavaScript spaghetti. The author did not use neither loops nor arrays. I squeezed the module to about 150 LOC. I also heard about a company, where programmers get bonuses for the lowest number of introduced bugs at the end of month. Their “top” engineers just do not write code. And there are dozens of such “success” stories over the Internet. If manager is dumbass, nothing will help from.

Nevertheless, sometimes it makes sense to get some numbers. It can be used as a signal that something went wrong. Maybe your teammate got family problems or burnout. Anyway, if a programmer usually produces X units of work, but have done only X/2 in the last measured period, it is time to figure out what happened.

I am not a manager, and I use metrics to estimate my own productivity. I often work totally without supervision and my collaboration with customers is only based on trust: “Do what you want to, but get it done.” So I need some metrics for self-control. And I discovered an optimal one (at least it works for me).

Meet the number of unit tests (NOUT). Unit test usually has constant complexity. It is a key feature. And it is obvious, that NOUT grows proportionally with program complexity. So you can measure productivity in NOUT/week or total progress in just NOUT. You can even try to estimate features in NOUT. It seems, it will be more accurate rather than story points. And if you start to pay your programmers for NOUT, they will just improve test coverage (it is a bad advise, don’t do that, they will find out how to fuck you up).

In my opinion, it is still far from perfect. But if it is used with wisdom, it could give you some advantage. If it isn’t, well... see the first paragraph.

It is not my regular article, it is the appeal for IT community. If you think, the thoughts stated below are worthwhile, please share them. To make it real we need efforts of whole community. Thus we need to make some buzz.

{
    "endpoints": {
        "login": "https://example.com/simple-auth/login",
        "logout": "https://example.com/simple-auth/logout",
        "register": "https://example.com/simple-auth/register",
        "testusername": "https://example.com/simple-auth/testusername",
        "changepassword": "https://example.com/simple-auth/changepassword"
    },
    "password": {
        "min": 6,
        "max": 255,
        "chars": ["a-z", "A-Z", "0-9", "!@#$%^&*()-_+=|\/?.,<>"]
    },
    "username": {
        "min": 3,
        "max": 30,
        "chars": ["a-z", "A-Z", "0-9", " _-."]
    }
}

Traversal is awesome thing, I believe that it is real killer feature of Pyramid web framework. However people usually don’t get it. They think it is too complicated. So I’m going to convince you in the opposite.

I assume, you know that Pyramid supports two URL handling methods: URL Dispatch and Traversal. And you familiar with the technical details of how them work (follow the links above if you don’t). So here I’m considering the benefits of Traversal, instead of how it actually works.

Pyramid is super-flexible framework, where you can do thing in the way you want to. Traversal is not an exception. To start working with Traversal, you just need to provide a root_factory callable, which accepts single argument request and returns a root resource of your web application. The root can be arbitrary object. However, to feel all power of traversal the root_factory should return a resource tree—a hierarchy of objects, where each one provides the following features:

• it knows its name, i.e. has __name__ attribute;

• it knows its parent, i.e. has __parent__ attribute;

• it knows its children, i.e. implements __getitem__ method in the following way:

  >>> root = root_factory(request)
  >>> child = root['child_resource']
  >>> child.__name__
  'child_resource'
  >>> child.__parent__ is root
  True
  

So that to build URL structure of your web site, you should build a resource tree—a bunch of classes, in fact. And that is exactly what usually confuses people. Is it overengineering? Why so complicated? Indeed, writing a dozen of routes will take exactly a dozen lines of code. Whereas writing a couple of classes will take much more ones.

However, the answer is “No”, it’s not overengineering. Traversal use resource tree for handling URLs, but the resource tree itself is not only used to represent URL structure. It is a perfect additional abstraction level which can encapsulate business logic. In that way the old holy war about fat models and skinny controllers (views in Pyramid terms) can be solved.

Resource also provides a unified interface between models and views. From one hand, you can build your models using different data sources: RDBMS, NoSQL, RPC, REST, and other terrifying abbreviations. And resources will make them work together. From other hand, you can use these resources in different interfaces: web (which Pyramid actually provides), RPC, CLI, even tests. Because test is just another interface of your application. And yes, using Traversal will make testing much more easier.

But what about URL structure? Using traversal is hard to start, you should build a resource tree. However these efforts will be rewarded in future. Because supporting traversal-based application is a walk in the park. For example, you have code that implements blog:

class Blog(Resource):

    def __getitem__(self, name):
        return BlogPost(name, parent=self)


class BlogPost(Resource):
    ...


@view_config(context=Blog)
def index(context, request):
    ...

@view_config(context=Blog, view_name='archive')
def archive(context, request):
    ...

@view_config(context=BlogPost)
def show(context, request):
    ...

Now, you can bind Blog resource to other ones, to add blogs into different places of your site. And it can be done with a couple of lines of code:

class User(Resource):

    def __getitem__(self, name):
        if name == 'blog':
            # Now each user has her own blog
            return Blog(name, parent=self)
        elif ...

From this point of view, resource with associated views can be considered as a reusable component, just like application in Django. You can also use mixin classes to create plugins:

class Commetable(object):
    """ Implements comment list """

class Likeable(object):
    """ Implements like/unlike buttons behavior """

class BlogPost(Resource, Commentable, Likeable):
    """ Blog post that can be commented and liked/unliked """

You can even make the trick, which I described in Obscene Python, i.e. constructing your resource classes on the fly using different set of mixins for each one.

And the last, but not least, Traversal is a right way for handling URLs, because it works with hierarchical structure which reflects URL. Whereas URL Dispatch uses flat list of regular expressions. So that, task like rendering breadcrumb navigation is trivial for traversal-based application, but it is hard as hell using URL Dispatch (in fact, it cannot be done without dirty hacks).

So if you are going to use Traversal, try also TraversalKit. This library is essential of my own experience of Traversal usage. I hope it will be useful for you too.

P.S. The article has been written in transfer zone of Moscow airport Domodedovo on my way from PyCon Finland 2014, Helsinki to Omsk.

I sighted increasing interest to my project Bash Booster last week. There is a lot of links, tweets, discussions over the Internet. So, I think I have to write this post to clarify what Bash Booster is and what it isn’t.

First of all, I am definitely not devop. However I played this role at the begin of last summer. This is how it happened. I finished my project, and had to wait while Thumbtack’s sales team was signing a contract of a new one. At that time, another team went into production and they needed help. Because I had nothing to do, I joined the team as a devop engineer. I had to write Chef recipes for AWS OpsWorks to deploy and manage application cluster. It was my first and last time I was using Chef.

Chef is awesome thing, but IMO it is too complicated. I spent a week looking into it. I think, it is mainly because of Chef documentation. I guess, it had been written by marketing-oriented assholes. I understand their goals, you can buy support, and they will make you happy. But if you decide to make it work by your own... Well, there is nobody to blame. Another reason is that Chef is written in Ruby (at least some of its components). The fact itself is not bad. However, in my experience, rubists don’t give a shit about backward compatibility. If Ruby is not your primary tool, get ready to shamanic dance. The most stuff does not work on Ruby shipped with your favorite distributive, because it’s too old. It is also very possible that it will not work on the most recent version, because this version breaks backward compatibility.

Despite all of these, I got working cluster after three weeks. And Chef have done its work perfectly. It is a good tool for devops, sysadmins, and everyone who works with dozens of servers. However, using it for provisioning single Vagrant virtual machine is overkill, it is like hunting a fly with sledgehammer.

When this project was done, my new one had been postponed, and I decided to go to vacation. But before it happened, I had joined to another team for a couple of days. They asked me to set up developing environment using Vagrant. I had written provisioning script using Bash, and went to booze trip into Kazakhstan.

In the middle of the vacation, I reviewed this script. It had been written in Chef recipe style. I didn’t think about it when I was writing it. But in fact Chef’s philosophy influenced me. So I decide to prove a crazy idea—write a Bash version of Chef subset.

As I wrote above, I am not devop, I am developer. I do not work with dozens of servers. It would rather an exception, when I had to. However, it is my normal routine when I have to setup single server for development purposes. It is also usual to me to share my virtual machines with teammates. So it is obvious, that I intensively use Vagrant. Because it is the best tool for such tasks.

Additionally, I do not like overcomplicated tools, which enforce end user set up various dependencies, write config files, and so on. And learning time is also critical to me. Ideally, it should be zero hours.

So, Bash is such tool. Everybody knows it, and it is installed everywhere. BTW, this fact have played Old Harry with Bash recently. I mean ShellShock vulnerability. But anyway, I decided to make a provisioning tool using Bash. So this is how Bash Booster was born.

It does not include any remote execution feature, because I do not need it. It does not try to cover all existent package managers, because I mainly use Debian-based and RedHat-based distributives. So support of Apt and Yum is enough for me. I also did not try, to unify these two ones. Because similar packages have different names in Apt and Yum, anyway you will have to check what manager do you use. So I applied JavaScript approach: test for feature, instead of version. That is why there are functions bb-apt? and bb-yum?.

I also like readability. Because of readability there is a lot of short functions of 1–2 LOC like this one:

bb-apt-package?() {
    local PACKAGE=$1
    dpkg -s "$PACKAGE" 2> /dev/null | grep -q '^Status:.\+installed'
}

I can use $1 variable directly, but assignment to $PACKAGE means that this function accepts single argument—package name. It is for self documenting. The black magic spell of the second line... Well, I do not want to see it in my final script. After a month I would have to find out what does it mean. And I would hardly reproduce it in the next script. So let it live in the library.

The main feature of Bash Booster is delayed events. It helps to avoid unnecessary operations. That is why sync module does not use rsync (I got a lot of criticism about it). It is not about coping files and directories, it is for triggering events when target and destination files are different.

In short, I solved my practical tasks developing Bash Booster. It is definitely not a Chef substitution. You can not manage a server park using it. It is not useful in writing universal platform-agnostic scripts. But if you need a little bit more than simple Bash script, especially when you need event driven Bash script, it will be helpful for you. Moreover, if you missed something, I will be happy to merge your pull request.

When you develop a library, which should work with a number of Python versions, Tox is obvious choice. However, I start using it even in application developing, where single Python version is used. Because it helps me significantly reduce efforts of documentation writing. How? It transparently manages virtual environments.

For instance, you work on backend, and your colleague works on frontend. This guy is CSS ninja, but knows nothing about Python. So you have to explain him or her how to start the application in development mode. You can do this in two ways.

The first one is to write an instruction. The instruction should explain how to setup virtual environment, activate it, setup application in development mode, and run it. In 9 cases of 10 it blows non-pythonista’s mind away. Moreover, writing docs is tedious. Who likes writing docs, when you can write a script?!

And this is the second way. You can write a script, which creates virtual environment, activates it, installs application, and runs it. But this is exactly what Tox does.

Here is how I do it. The following tox.ini file is from the project I am working on now. It is a Pyramid application, which I test against Python 3.3 and 3.4, but develop using Python 3.4.

[tox]
envlist=py33,py34

[testenv]
deps=
    -r{toxinidir}/tests/requires.txt
    flake8
commands=
    nosetests
    flake8 application
    flake8 tests

[testenv:dev]
envdir=devenv
basepython=python3.4
usedevelop=True
deps=
    -r{toxinidir}/tests/requires.txt
    waitress
commands={toxinidir}/scripts/devenv.sh {posargs}

Take a notice on the section [testenv:dev]. It launches devenv.sh script passing command line arguments, which are not processed by Tox itself. Here is the script:

#!/bin/bash

test() {
    nosetests "$@"
}

serve() {
    pserve --reload "configs/development.ini"
}

cmd="$1"
shift

if [[ -n "$cmd" ]]
then
    $cmd "$@"
fi

And here is an example of the manual:

1. Install Tox.

2. To run application use:

   $ tox -e dev serve
   

3. To run all tests using development environment:

   $ tox -e dev test
   

3. To run single test using development environment:

   $ tox -e dev test path/to/test.py
   

4. To run complete test suite and code linting:

   $ tox
   

That’s it. Pretty simple. I copy it from project to project and my teammates are happy. You can even eliminate the first item from the list above, using Vagrant and installing Tox on provision stage. But there is a bug in Distutils, which breaks Tox within Vagrant. Use this hack to make it work.

It is always easy and fun to do something, if you have right tools. Writing tests is not exception. Here is my toolbox, all things at one place. I hope, the following text will save somebody’s time and Google’s bandwidth.

Here we go.

$ flake8 coolproject
coolproject/module.py:97:1: F401 'shutil' imported but unused
coolproject/module.py:625:17: E225 missing whitespace around operato
coolproject/module.py:729:1: F811 redefinition of function 'readlines' from line 723
coolproject/module.py:1028:1: F841 local variable 'errors' is assigned to but never used

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
coolproject                  20      4    80%   33-35, 39
coolproject.module           56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%

[nosetests]
verbosity=2
with-doctest=1
with-coverage=1
cover-package=coolproject
cover-erase=1

$ nosetests
tests.test1 ... ok
tests.test2 ... ok
tests.test3 ... ok
Doctest: coolproject.function1 ... ok
Doctest: coolproject.module.function2 ... ok

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
coolproject                  20      4    80%   33-35, 39
coolproject.module           56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%
Ran 5 tests in 0.021s

OK

[tox]
envlist=py33,py34,pypy

[testenv]
deps=
    nose
    coverage
    flake8
commands=
    nosetests
    flake8 coolproject
    flake8 tests

So, a product owner created a ticket with the text from headline of the article. You got it into sprint and started to work on it. How will you implement it? Usually we think that e-mail notification is something asynchronous. We can send it anytime. Commonly at night, when our server is under low load. User will read it in some time, but not right after you have sent it. It was true before we have gotten smartphones. Nowadays most of e-mails come instantly with loud notification sound. And user will not pleasured to hear this at sleep time. Therefore I think it is not a good idea to send e-mails at night. So next time you will implement such feature think about it.

P.S. Thanks to LinkedIn for this article. I am very happy to get notifications at 3AM when someone from my connections is celebrating a work anniversary. That is completely what I need to have a good night. And guys... I pointed my location in the profile. It would help to detect my time zone.

Since I started developing in Python, I use Setuptools in each of my projects. So I was sure that this approach is obvious and nobody needs an explanation what benefits it brings. I think, it happened because the first thing I’ve learned was Pylons web framework. There was no other way developing project rather than using Setuptools. However I was wondered to know how much people develop applications without packaging and get troubles, which already solved in Setuptools.

Let’s consider a typical application. It usually consists of single package that includes a number of modules and subpackages:

MyProject/                  # a project folder
    myproject/              # a root package of the project
        subpackage/
            __init__.py
            module3.py
            module4.py
        __init__.py
        module1.py
        module2.py

If you are going to use Setuptools, you have to add at least a single file at the project folder—setup.py with the following contents:

from setuptools import setup

setup(
    name='MyProject',
    version='0.1',
    description='42',
    long_description='Forty two',
    author='John Doe',
    author_email='jdoe@example.com',
    url='http://example.com',
    license='WTFPL',
    packages=['myproject'],
)

This script adds metadata to the project and tells Python how to install it. For example the following command installs project into Python site-packages:

$ python setup.py install

...and this one installs it in development mode, i.e. creates a link to the code instead of copy it:

$ python setup.py develop

If you ever developed a library and published it on PyPI the code above should be familiar to you. So I’m not going to discuss why do you need Setuptools in library development process. What I’m going to consider is why do you need Setuptools in application development. For example you develop a web site. It should work on your local development environment and production one. You are not going to distribute it via PyPI. So why do you need to add extra steps of deployment—packaging and installation? What issues Setuptools can solve?

#!/usr/bin/python

import os
import sys

# Makes ``myproject`` package discoverable by Python
# adding its parent directory to import path
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path = [root] + sys.path

from myproject import module2

from . import module2
# You will get
# ValueError: Attempted relative import in non-package

$ /path/to/myproject/dosomething.py

$ sudo ln -s /path/to/myproject/dosomething.py /usr/bin/dosomething
$ dosomething

setup(
    # ...
    entry_points="""
    [console_scripts]
    dosomething = myproject.module1:dosomething
    """,
)

[group_name]
entry_point_name = package.module:object

import pkg_resources

plugins = {}
for entry_point in pkg_resources.iter_entry_points('myproject.plugin'):
    plugins[entry_point.name] = entry_point.load()

[paste.app_factory]
main = myproject.wsgi:get_application

setup(
    # ...
    install_requires=['Pyramid', 'lxml', 'requests'],
)

$ pip intall -U pip setuptools wheel

#!/bin/bash

APPLICATION="MyProject"
WHEELHOUSE="wheelhouse"
REQUIREMENTS="${APPLICATION}.egg-info/requires.txt"

python setup.py bdist_wheel

mkdir -p "${WHEELHOUSE}"
pip wheel \
    --use-wheel \
    --wheel-dir "${WHEELHOUSE}" \
    --find-links "${WHEELHOUSE}" \
    --requirement "${REQUIREMENTS}"

cp dist/*.whl "${WHEELHOUSE}/"

$ pip install --use-wheel --no-index --find-links=wheelhouse MyProject