Initially he didn't know a lot about transactions, so he researched them in depth. A quote by Christophe Pettus: "transaction management tools are often made to seem like a black art".

He moves from the database (postgres and sqlite) to the interface (psycopg2 and sqlite3) to the framework (django).

The classical approach in django is:

Code development

You create models.

Deployment

Tables and columns are created with syncdb.

Runtime

Models and db tables are populated.

This means that models are pretty much static. There is no way to modify them at runtime based on user interactions. You can get something working with for instance hstore in postgresql (see the postgresql talk).

His usecase is for medical forms. The contents of those forms should be able to be defined inside the system. There are strict processes for installing medical software, so you cannot just release a new version with a new field. So you must get it to work at runtime.

The solution could be to use dynamic models, models created at runtime. Sometimes configuration by subject matter experts is better than code customization by developers. Also, dynamic models reduce the number of deployment cycles.

He has some criteria:

• Performance.
• Querability, which means the standard django query stuff should work.
• Django standard tool integration (admin, cache, and so).
• Supported DB backends. If possible, support all django DB backends.
• Complexity/maintainability.

There are a couple of possible solutions:

Entity attribute value (EAV)

Colums are stored in separate table rows. Instead of a table having attributes, a table has an attribute table with values. There are at least two apps that provide this. The performance is a problem here.

Serialized dictionary

For instance one of the Django JsonField apps. A lot of what is normally database work is now moved to the application. You'll really have to create custom logic in your app to take care of it.

Runtime schema updates

Update models at runtime with syncdb of some South functionality or Andrew's new schema migrations for Django work. There are a couple of apps that do this. He also created his own one. The best one seems to be django-mutant.

Database-specific solutions

Hstore, django-nonrel. Drawback: it doesn't work with all database backends, of course.

In the end, the runtime schema updates approach looks like the best one.

For more reading: https://code.djangoproject.com/wiki/DynamicModels

They grew from a two-person company to a 70-person one in two years. Central to that growth were Django and google app engine.

Scalability means both load scalability and functional scalability. You also have to deal with organizational scalability and geographical scalability if you want to grow your organization.

If you've got continuous deployment, you've got stable servers. You make big changes in small increments.

Continuous deployment forces you to do many good things:

• Good tests.
• Repeatable build.
• Well-configured identical machines.
• Automated deployment.
• Migrations and rollbacks
• Etc.

Lots of good things. But let's compare it with lion taming.

Originally, lions were beaten into submission, confused and kept in line with whips. Likewise you'll be beaten if you dare to touch the production machine as it might break.

Now lions are understood better. Conditioning, behavior/signal mapping, reward and trust are the methods now. We understand that deployment is hard. We have behaviour/signal mapping with code/test/green/deploy. Etc.

• Continuous deployment: everyone is responsible. Everyone deploys. You automatically learn. Everybody uses the same environment locally for test deployments. The same as on the server.

• Testing is core. Slow tests are killing. Fast tests. And all types of tests: unit, functional and acceptance tests. Also automatic code checkers. The light must stay green. Quality must stay high, also test quality.

• You need a repeatable build. And it should include not just code, but also configuration and infrastructure. And... always follow the pipeline.

  Even in emergencies, follow the pipeline. Peer review, tests, and then the deployment. Don't do manual steps.

• Rollback. You must be able to switch back to the previous version.

  You can take a canary approach. Canary in the coalmine. Show the new version to a few users. "User testing" in a sense.

  Rollbacks in databases; keep it backwards compatible. Never drop columns, for instance. (After a long time, you can remove them safely, of course).

• Small changes. Frequent releases mean less risk: if something breaks, you know where to search.

Some tips:

• Split your stuff in components. A component is something that has a good API and that can be switched out for a different component. It can also be separately deployed.

  This helps with testing, too.

• Rehearse releases. Get very good at them!

• You need good infrastructure. You must manage it and test it good.

• Keep all environments equal. Use vagrant.

• Automate everything! And if it is not possible: document it. But know that that's something that's not quite correct yet!

Principle philosophy: a way to discuss our rules and beliefs that govern our actions. He tells it from his personal experience.

His parents wanted to raise him as a good person. So they thought him good principles (like don't be a quitter, don't steal, etc). This is quite black/white though. We are all more gray/gray.

What about the question "how can I be a good programmer"? Programmers use logic, which sounds black/white again: write tests, don't repeat yourself. Sigh.

Talking about things like this is impossible without Immanuel Kant. He differentiates between reason and instinct. If "be happy" were our life goal, we'd just follow our instincts. So what is reason for, then, apart for doing good? Reason has to do with moral. There are three ways of looking at "doing good":

• Duty. Good things can come from duty. Duty can also lead to non-good things, though. Hm, so this is not it.
• Make a difference between the goal and the outcome. The outcome might be bad even though the goal could be worthy.
• Universal lawfullness. Only do something if you know that everybody thinks it is a good idea.

Does this help with a question like "is testing good"?

Gandhi said that a man is the sum of his actions.

In a sense we are the sum of our experiences. So increase the amount of experience that you have. Either have the experiences yourself, or share them like on this conference. Everything looks different from the trenches: learn from eachother.

Some lessons he learned from a little baseball league experience (where he sucked) as a kid:

• Swing for the fences. Aim for a home run. It allows you take great risks (because you have great goals). It motivates you.
• Set reasonable goals, too. Incremental intermediate goals. Those intermediate goals help you progress.
• You suck... and that's totally OK. You're not good at everything. It gives you a different perspective. And you can still give it your best. Also to that almost-unused old project that you get a bug report for.

Some take-aways:

• Build a strong foundation of principles.
• One size doesn't fit all
• Learn from your experiences and share them.
• Build a great network.
• Ask all the right questions.

Open source is great. It is absolutely amazing.

We live in a multi-dimensional world, though it is often presented otherwise.

Some present a simple line-based worldview. Bad-Good for instance. Where do you want to be on the line? Republican-democrat? Ruby-Python? Foreigner-native? Once you think along those lines (...) you tend to start thinking in opposites.

This is the basis for many invalid world views. Just draw a line, cluster according to your preference and you're ready. Linear concepts are not useful. The issue is polariation. In a one-dimensional world, there is no room for complexity.

What about a Venn-diagram based worldview? It allows for a bit more subtlety, but there's still a line on the outside...

The open source world has a lot to teach the rest of the world. It is focused, mostly, on outcomes and results. But it is not representative. It is not even representative of the tech industry generally. In tech, 20% are women, in open source it is more like 2%, for instance.

And... we need diversity! The biggest resource in open source is people. So you'd rather not exclude many people. The most common diversity areas, to give you an idea:

• Ethnicity
• Religion
• Gender
• Culture
• Socio-economic background

Diversity is desirable because each individual is limited. We are all good at some things and bad at others. A group can solve a bigger range of problems And you don't want the group to be too homogeneous.

Typical open source projects will tend to focus on the actual programming and it'll ignore technical writers, designers, training, etc. Django stands out with its documentation. But Python's documentation isn't that good. There's no real emphasis on it in the current Python team. If we don't broaden our community with different skill sets and roles, we'll fall behind. Python is poised to be the #1 language of choice, but we need to improve some things before that can happen.

We ought to involve the community more as open source projects. We should run our projects more professionally. Be more open to involve all of the community more.

We should not accept it anymore to have to read through half-finished documentation and having to fall back to reading source code. "But that takes time to rectify". Well, yes. So involve more people. Get more people with more diverse skill sets to help. Perhaps you can then focus on what you're good at.

It is up to us all. The python world does have an awesome community. But we might just be a bit too smug about how wonderful we are. We should not get complacent and we should keep aiming at increasing our diversity.

Russell is a Django core dev.

Class based views were introduced two years ago, but they weren't greeted with universal acclaim. So he's here to clear up the mess and hopefully make it all more clear for everyone.

She works for Red Hat on http://freeipa.org, on identity stuff for Linux.

Note: see her website for instructions and code examples.

Say that your pointy haired boss (or customer) asks you to make an internal web app with all the buzzwords.

So you can't use regular django auth, you'll need single sign on. Luckily since Django 1.5 you can have custom user models, so it'll fit with all your external requirements. One or two pieces of MIDDLEWARE_CLASSES and AUTHENTICATION_BACKENDS later and you play nice with the external single sign on. Django can be a team player.

Webserver? You'll probably have to use apache. So the environment can be kerberos+apache. Add mod_auth_kerb for kerberos support. Add a "keytab" (making sure it is chown'ed to apache).

There's a difference between authentication and authorization. Authentication is "just" logging in, authorization is what you're allowed to do. You'll have to connect to LDAP for that to ask which group(s) the user is a member of.

Setting up your own kerberos environment (for testing) is a pain. Unless you use a ready made vagrant box for it. Instructions are on her website.

Django conferences have a tradition: there's an external luminary that gets to give a critical talk on Django. His talk won't be that. He's not external either: he wrote two scoops of Django together with Audrey (see also my review).

Being critical is sometimes easy. Just bash class based views, for instance. Bashing is easy. A rant like Zed Shaw's is fun, but he's not asked because of his rants, but because of his contributions (like books).

Similarly, Django delivers working stuff and that working stuff makes a lot of our work possible. So here are some good points about Django:

• Django is everywhere. So many people and companies use it.

• Django is powered by Python. Pep8, python is beautiful. And there's the import this zen of python that we use all the time to steer others and ourselves in the right direction.

• Django's API wins. It is understandable. No weird names: templates, views, logging, sessions. Django projects also have understandable structures. If there's no views.py or models.py or templates/ directory, you know someone messed something up.

  Fat models are great. Just put your business logic all on your models. They do get big this way, however. You can make a separate module with helper functions you call in your model. Just call it with a model attribute. This way you get a reasonably small models.py and another file with very testable small functions. Win!

  The API is clear and logical. We're not fighting about architecture, we are getting things done.

• Django views are simple callables. (Even class based views).

• Django is awesome at deprecation. Code often just keeps working fine for multiple Django versions.

• Django has lots of features. For instance, Django's admin is awesome. This is what you use to sell Django to others.

  Tip: don't try to use the admin with a nosql database. Just build something from scratch, that's easier than trying to get it working and especially keeping it working.

• Django's full stack is awesome. Real projects are being done in unextended python+django. No third party packages. Not everyone goes to conferences and not everyone knows everything on pypi. You can get a lot done with just Django (though external packages help a lot!).

  Django is also python. There are over 30000 packages! Even if only 20%

• Documentation. Django set the bar for others.

  Truth be told, some other projects have better documentation now. Django set the bar and others followed. They're playing by our rules :-)

• Django is humble. We have a tradition of invited critical talks. They shape the community, they shape the core committers.

  Here are two critical talk summaries I have:

  • Martijn Faassen 2011
  • Kenneth Reitz 2012

  Good criticism is good. They got a lot on their book. That was hard, but the book is much better for it.

• The django community is generous. There's almost an unwritten rule: "the more you help people in the Django/Python community, the more the community helps you".

  They recieved a lot of help with their book. They also helped others with free books in case they couldn't pay for it. But that required an email to ask for it. Note that Daniel and Audrey did ask for something in return: either buy the book later, donate to charity or help someone.

  It worked! Somone gave a free guitar course to someone else. Someone bought a homeless person a dinner. People did projects for schools/churches/whatever. Contributions to open source software.

  It worked! People did good work! Lots of small local positive actions. All over the world.

Call to action: be awesome. Make the world a better place.

Sorry if I mangled any of the names, that's the hardest part of blogging lightning talks. Many don't show their name long enough :-)

A plane flew over (noisily) at the start of his presentation. He put our work in perspective by saying that that was a 80 ton plane and that we're just building websites :-)

He runs multiple sites with a common business model: accounts with plan subscriptions. So there's an obvious need for a generic account billing application.

The app should not be too specific, as that limits your business flexibility. Also it should not be too generic: you'll end up with an architecture from hell that way. And there's the billing as such: you need to pay close attention to security and so. Hard problem!

What he's making is django-plans for keeping track of the billing data, the plans, etc. And django-getpaid as payment processing app.

Jérémie is a frontend developer and Marc does the backend.

Address/state handling and content rendering are the two main challenges.

He shows us three kinds of (more or less) open source projects.

Metaclasses are a handy feature of Python and Django makes good use of them.

When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class.

Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance.

As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class.

Did you know that you can create classes dynamically? See for yourself:

>>> name = 'ExampleClass'
>>> bases = (object,)
>>> attrs = {'__init__': lambda self: print('Hello from __init__')}
>>> ExampleClass = type(name, bases, attrs)
>>> example = ExampleClass()
Hello From __init__
>>> type(example)
<class '__console__.ExampleClass'>

So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, that's what we saw in the first Django form example!

Now, what is type exactly? It is a class that creates classes.

This also means we can subclass it! The most useful thing to override in our subclass is the __new__() method. The __init__() method creates instances from the class, the __new__() creates classes. So again we can modify the name and/or the attributes.

How do you use it in practice? Normally you'd set a __metaclass__ attribute on a class. This tells python to use that metaclass for creating the class. The same for subclasses. This is how our Django form classes use the metaclass specified in Django's base Form class.

Django uses metaclasses in five places: admin media, models, forms, formfields, form widgets. Grep for metaclass in your local django source code once to get a better feel for how Django uses it.

Note on python 3: it uses a slightly different syntax for specifying metaclasses. So Django 1.5 uses six to support both ways in a single codebase.

Warning: don't overuse metaclasses. They can make code difficult to debug and follow. Use Django as a good example of how to use metaclasses. Django saves you a lot of work by using metaclasses in a few locations.

See https://github.com/inglesp/Metaclasses

Nice way of giving a presentation, btw. Some sort of semi-interactive python prompt. The software is online at https://github.com/inglesp/prescons

He started with a little history lesson. The sea battle of mobile bay. The admiral (Faragut) ordered the ships straight through the minefield (called "torpedoes" at the time). "Damn the torpedoes, full speed ahead". And it worked.

What does this to have to do with Django? Well, "damn the torpedoes, full speed ahead" feels a bit like how rapid prototyping feels afterwards. He's often involved with hackathons. Lot of quick coding in limited time with a lot of people. He learned a lot about his tools that way (and he often used Django).

There's a time to make a distinction between production and prototype. Sometimes it is better to just try something with a prototype. Throw-away code.

Aaargh! Throw-away code?!? We never throw code away. But it is something we must learn. It is good to let go once in a while. Let your code go. It isn't yourself, it is just some code.

The danger is that prototype code is put into action as production code. With some work, this danger can be prevented.

What about Django? Django is the best for prototyping. For rapid prototyping, Django is better than micro-frameworks like Flask that might seem better at first glance. Here are some reasons:

• Django was build for rapid prototyping. It originated at a newspaper! 24 hours to build something.

• It is flexible. It was build to bend. He can prepare something for the other people programmign with him and get them going and still keep the code in reasonable shape.

• Us. The strength of Django is the community that supports it. Stack overflow questions and answers. The django websites. Books like two scoops of Django (see my review). That's not something you have with many other frameworks!

  Tip: read especially chapter 2 and 3 of the two scoops book.

  One thing he'd add to the book is stuff like fabfiles and makefiles. Handy for rapid prototyping.

Use stuff that's available. For instance Django's staticfiles app for grabbing together all the css/js/png. Whether it is in one directory or split out over multiple apps. It also helps with production.

Also look at brunch for setting up your javascript app's structure. It works well with Django.

Deployment: you need to show your prototype. Heroku is very quick for prototypes. (He mentioned that they have a data center in Europe now in case you need it).

Deployment: use chef. Lots of recipes. You could also use Salt if you're more into Python. Also lots of stuff available. Both take a while to learn, but it is a very good investment.

Configuration management is an extremely useful skill. Do it well.

Tastypie is the quickest way to get a REST api out of your Django. It is the best for rapid prototyping. Another good one is django-rest-framework. It will take a little bit longer to set up, but once done you're working with actual Django views. And django-rest-framework's browseable API is very helpful when you're working with a couple of others

Social auth connectors: everyone makes one and there are way too many half-working ones. He's got two that he can recommend. django-social-auth is very complete. The other is django-allauth for when speed is important for you.

If you don't want to play fair to others during a hackathon: use celery. It is very unfair to use celery, python and Django. The combination with Django is pretty OK to set up. You can do a lot what others cannot do easily. So use it for rapid prototyping. (Regarding setting it up: there are good chef recipes for it).

TEST. Yes, even during a hackathon. He doesn't advocate full test driven development. It is a balance. But the errors that kill you during a hackathon are the errors you make twice. So, for instance, test that all your views simply return a 200 Ok. This already helps prevent a lot of problems.

Look at AngularJS. Even if you don't use the framework itself. Why? It has a great javascript test runner. Good for testing while rapid prototyping.

Tip: subscribe to the postgresql weekly newsletter that Craig makes.

Why postgres? A colleague described it as "it is the emacs of databases". There's just so much available inside postgres.

The problem is Django: it treats all databases the same. It doesn't prefer one over the other. It doesn't give special treatment. Look at all the types that Christophe mentioned yesterday: Django only supports a few of 'em. Likewise indexes.

For instance postgresql's Array type. Django doesn't support it, but it'd be perfect for for instance a list of tags on a model. For many of these types, also for the Array type, you have django apps that add support for them.

Great: hstore. NoSQL in your SQL. A key/value store in your SQL. They use it inside Heroku a lot: it scales fine and works fine. To use it in Django, use django-hstore. Add a data field as hstore to a model and suddenly you can do my_object.data = {'key': 'value', ...}!

Queuing: most people use celery. Postgres is a great queue. There's a celery backend called trunk for it.

Postgresql has great text search. You do need to do some setup in your models, but then it works fine. You'll have to read the docs, though.

Indexes. Many types. So it can be a bit of a mistery which one to use. Btree is the normal one. Generalized inverted Index (GIN) is for multiple values in 1 column. Good for array/hstore types. Generalized Search Tree (GIST) is for full text search, shapes, postgis.

Geospatial: just use geodjango. It uses postgresql/postgis's great geospatial stuff.

Tip: look at django-db-tools, for instance for its read-only-middleware that makes your site read-only (for maintenance, for instance).

Django 1.6 has persistent connections, but the current 1.5 doesn't. It can shave a whole lot of the rendering time of your pages if you have some sort of connection pooler! If you want the 1.6 functionality now, you can use for instance django-postgrespool. This really saves a lot of time.

Summary: postgresql is great, Django's ORM is pretty good. And you can extend it.

(In response to a question: never put session data in your database, it is a good way to kill the database.)

Here's the link to his presentation.

He worked twisted on. twisted And people tend to talk about subjects that are almost antithetical to how Django does things. The thing that he does different from Django is that he's not using a single data source...

Once a database gets really really too big, putting multiple databaseservers next to eachother doesn't really work. You slowly start to get into expensive Oracle territory.

How he set it up now is what he calls a fractal architecture. The whole accepts requests. The parts of the whole acccept requests. The parts of the parts accept requests. That's why he calls it fractal. You could also call it sharded, but that has a bad name: it is something you do when nothing else works.

The way he looks at the architecture is SMTP. Email. Simple.

He prefers SQLite. Simple and included in the python standard library. Sure, you can use postgres but you'll need a VM to re-create the same environment locally as on your production machine. SQLite is the same everywhere.

In fact, he uses Axiom: an object store on top of SQLite. (Note: he is trying to write documentation for it at https://github.com/lvh/axiombook).

Another advantage of sqlite: it is easy to scale down. There's not much lower you can go than import sqlite3! If you want to use postgres, remember you must install it on each and every part :-)

Important: almost nothing is as fast as a local sqlite store, especially when it is reasonably small and fits mostly in RAM. Just look at the regular comparisons of access time for L1 cache, L2 cache, RAM, SSD, LAN, spinning rust, internet and so. So if you have a local database on an SSD with quite some RAM, it'll blow a network connection to some remote database out of the water.

But... some things don't fit locally. You have to search everywhere, for instance. There are three basic solutions:

Duplication

You could duplicate the data over all the parts, but that doesn't work if the data is big.

Sharding

Sharding will only work reasonable if the data itself, by nature, is sharded. Sales data per region, for instance.

Separation

Separate data for separate calculations in separate (local) stores. This is what he uses.

He mentioned paxos and raft (pdf), but I don't remember what for.

Many people think that reusable apps don't work: there's always something you need to change or modify. Honza is going to talk about his experience with ella, a django CMS.

He advocates using model inheritance. from ella.core.models import Publishable and then subclass your specific model (YoutubeVideo, for instance) from it. That Publishable has most of the basic CMS functionality. That way you get most of what the CMS needs for free and you still can extend it.

Showing the new model? You can use different templates easily. render_to_string() and friends accept a list of templates. So you can give it ['publishable.html', 'youtubevideo.html'] and so, using templates named somewhat after the model. This way you can re-use basic templates, but modify them if you want, just by providing a specially-named template. No code changes necessary.

They're using Redis to collect information from the Django database on publishables. This way you don't have any problem with Django's database's behaviour of focusing on a single kind of model at a time.

They also use django-appdata for storing extra data on existing Django models. From the pypi page: extandable field and related tools that enable Django apps to extend your reusable app. Through a registry you can add for instance tags to an existing model. To actually see the field in the admin, you do have to make a new ModelAdmin for that . Django-appdata is a hack, but sometimes hacks are very useful.

Through ella.core.custom_urls they even managed to add URLs (like a URL for adding "+1" functionality) to arbitrary models that support it (through django-appdata).

Warning: with great power comes great responsibility. All this is powerful, so it is easy to make a complete and utter mess out of it. Perhaps it is better to convince the customer to forgo a feature?

Warning: keep the defaults sane. Nice to use Redis to make querying quicker and simpler, but you just forced every developer to have Redis installed locally.

Warning: premature optimalization is the root of all evil. Likewise with extensibility. You normally don't need to make an app extensible. But don't close the door to extensibility. Add it when you need it.

Marek works at SetJam: "We came to Django for the views, but stayed for the ORM". Django's ORM is pretty much in the sweet spot. SQLalchemy in comparison is less nice, having to learn a non-sql, non-pythonic language.

At SetJam, they have what they call a backend and frontend. The backend collects data and stores it in the database, the frontend spits it out, mostly via feeds.

They started out with one single big database, but that was hard to optimize. Many backend servers would write to the same database and the frontend server would read from it. Hard to optimize.

Next they added a database slave for reading. That was before Django's multi-db support, so they had if/elses in their settings files based on environment variables.

After Django's multi-db support, they could really support two databases and refer to them in the code with 'DEFAULT' and 'SLAVE'.

Later on they splitted up the database even more. What goes where is handled by two custom database routers: a "MasterSlaveRouter" for the master/slave distinction and an "AppRouter" for shuffling some apps' data to certain databases.

Tip: look at https://github.com/jbalogh/django-multidb-router, especially for the handy decorators (@use_master, for instance) it provides.

At a moment they had problems with Django's transaction decorators: they only work with the default database. They had to call the actual code and pass it the right database.

Similarly, South doesn't work very automatically with multiple databases. South's ticket #370 is still open after three years. He hopes he can get a fix into the new south-in-the-django-core code.

He showed a code example that looked pretty OK. Then he showed what needs fixing to get it to work reliably with multiple databases.

Multidb is awesome, but...

• It needs more documentation.
• Full support for multidb in schema migrations.
• It needs better debugging tools (whiny transaction decorators).
• Attributes like _for_write should be more clear. They're pretty important, but the underscore looks like it is unimportant. (Comment: a core dev discussed with him during the questions; he thought this wasn't necessary).

Sorry if I mangled any of the names, I took a photo of the lightning talk submission form and tried to decypher them :-)

Andrew Godwin attempted to raise 2500 pounds for inclusion of south in Django core with kickstarter. It worked. In fact, he raised 17952 pounds!

Why does South need to be replaced by a new version inside Django itself?

• It started 5 years ago, so there's 5 years of learning done in that period. Some things that made sense at the time aren't the best decision now.
• There's poor support for VCS branching.
• The migration files are huge.
• Migration sets get too large. There are projects with 1000 steps!

The inside-django solution has two parts. The actual migration code and a separate backend. So if you want a different migration engine, you can probably reuse the backend code with its support for multiple types of databases.

The new migration format is more declarative instead of imperative like it is now. This makes them smaller. It also allows you to compute the end result in memory and apply one single migration.

Migrations will have a parent. So you won't have a problem with 0003_aaaa and 0003_bbbb migrations that halfway bite eachother. If a merge can be done automatically, fine, otherwise south/django will warn you.

Squashing will be added. You can squash a set of migrations together so that you can start from one new starting point instead of needing to go through the entire list of migrations.

One thing to watch out for: the Django field API will change a bit because the migration code needs to know how to re-create a field. Watch the django developer mailinglist if you're interested.

Read his blog at http://www.aeracode.org/ if you're interested in the details of everything he encounters.

Jacob Burch hopes you can learn from him if you're new at contributing to open source. He won't cover virtualenv, git, django's core code structure. And also not what to get involved in. What's this talk about? About you if you have something you want ("a pony") to get into Django core.

You are initially probably going to be a bit afraid. Jacob showed a couple of quotes about people that were initially not quite sure/certain when committing to Django. Then he showed the names of the people those quotes came from: they're now all core committers :-)

Two balances you have to keep in mind:

• You should be both pro-active and patient. This is a tough balance to strike. If you manage it, it helps a lot.
• You should be both confident and humble. Be humble, but be convinced of your idea. How to help here? The best thing is to run all the tests. It will give you confidence that your solution works (if it does). And it'll make you humble once you realize all the end cases that Django (and thus your fix) needs to support.

There are three broad categories of contributions:

Bug fixes

Start with a test condition. Something works or it doesn't. A test that demonstrates an issue is worth 20 emails.

Major contributions

Do your homework. Search trac, search the django developer mailinglist, become familiar with the code you're proposing to change. You need a go-ahead beforehand, so discuss it on the mailinglist.

Minor additions

Treat it as a major contribution. Only a beforehand go-ahead isn't needed here.

(Jacob did some live coding, trying to get a push into Django. In the meantime, he continued with the presentation by showing himself on video :-) )

Some do's/dont's when mailing about something:

• Don't communicate entitlement. Don't focus only on your own needs.
• Communicate patience. Accept that this is the start of a conversation.
• State the problem clearly.
• Show confidence: propose a clear solution. This really helps the core devs, as they have a clear proposal to work from instead of having to come up with something themselves. Creative energy is expensive energy.
• Show your homework. Ticket numbers, list potential downsides/drawbacks.
• Show humility. If you're unsure of an aspect, just ask.

Code is important, but most of the effort will probably be spend in discussing it. That said, here are some code related suggestions:

• PEP8, unless it is consistently ignored on a certain point. Stay consistent locally.
• Respect existing style.
• Comments are your friend. Don't comment too heavily, but make sure that anything unusual is explained.
• Get some peer review before submitting.

Repeat to yourself: you are not your code. Your ego is not on the line. Separate yourself from your code. Humility is really important. Your patch might not get accepted. You might get negative feedback. Don't take it personally. Your code is not yourself, even though it might feel like your own baby.

If it is not getting reviewed: remember that core devs are busy and might not have had time to review it. A bit of persistence is important, but don't irritate people. Tip: get to know people that can commit on conferences or at sprints. That helps.

Once you do get feedback: iterate quickly and get back quickly on the feedback, otherwise the core dev has to load everything back into their head.

Django is a javascript-agnostic web framework. Nothing is built-in so you can be up to date all the time. Javascript development moves very quickly.

The basic approach is to include some custom inline javascript in the html pages. It quickly leads to illegible code that's hard to work on and hard to distribute.

Javascript has frameworks, too. They give your application structure and take work off your hands. This is the advanced approach. It includes several parts:

• Communication with the server (REST api, websockets).
• Application building: combining and minimizing files.
• Static files management.
• Javascript improvements: coffeescript and so.

What Przemek Lewandowski needed was a powerful javascript framework, coffeescript, testable code, js code minimization and fingerprinting for avoiding caches. And also rapid REST API development.

Javascript framework

They started with backbone, but it wasn't enough. They added marionette to backbone, but it still wasn't good enough. There's a lack of a binding mechanism; there are no reusable views; models are poor. AngularJS and Ember are better.

Coffeescript

It is controversial, but it helps to write code faster and use less code for it. It performs as well as javascript as it compiles to javascript. They used requireJS for painless coffeescript integration. Requirejs allows for modular code and gives you both a builder and an uglifier.

Building javascript apps

In the end they used django-require instead of django-compressor and django-pipeline.

REST api

Piston isn't really maintained anymore. Tastypie is reasonable, but django-rest-framework is the nicest one. It uses class based views, so it saves you a lot of work (even though still being very customizable).

Static files management.

Django's built-in static files management is good. And you can add extra "storages" to it to get django to store the static files in the cloud, for instance. django-require can be plugged in, too, to add a fingerprint to javascript files to ensure the latest version is always used.

There's more, like Bower, a javascript package manager. He didn't look at this yet. (Note by Reinout: look at http://blog.startifact.com/posts/overwhelmed-by-javascript-dependencies.html for a starting point)

Ania Warzecha researched recommendation systems. Recommendations means estimating ratings or preferences for items a user hasn't seen yet. For example books or movies you might also like based on earlier purchases.

There are three kinds of recommendations.

• Collaborative recommendations. Mostly created based on actions from other users. Which books are often bought together, for instance.

  Simple to implement, but can be slow for big datasets. And doesn't work well on new items and/or new users

• Content-based recommendations. Looks for similar items.

  Fast and accurate, but tends towards over-specifications regarding needed data.

• Hybrid methods. Combining them.

A case study: a Polish car parts website. You normally don't log in there, you just want a part. So older purchases aren't available. They did have a lot of parts and data, so they started with content-based recommendations.

They mixed in some basic user actions. 0=didn't buy, 1=browsed, 2=bought. Later on more elaborate, like points for items found through searching or items placed on wishlists.

They used Redis for its quick addition of user actions, simply pushing an additional score to an item which then gets added in the database.

One thing they needed to do was to merge session keys after a user logs in, merging the before-login session with the logged-in user's session. They didn't want to lose data collected till that point.

Now on to figuring out similar users. Common techniques are Euclidean distance, Pearson correlation and cosine similarity. But the problem was that it was slow. So they made an intermediary cache table in Redis.

Some conclusions:

• Redis is good for fast storing and painless calculations.
• Content-based recommendations are good for big datasets.
• Keep all the data you can keep.

(See also last year's talk)

Database agnosticism: write once, run on any database. A critical selling point for Django: it runs on many databases. But for others, it is bad. You pay a performance hit for not using database-specific features. So once you have made your choice, really use that database.

Here are some examples of good special things available in postgres.

Everyone should be paranoid when processing payments. The client, the programmer, everyone.

He works on Firefox OS and more especially the marketplace ("don't call it an app store"). The marketplace is powered by Django. And of course it accepts payments. And of course it is open source (even the presentation is on github).

Btw, they have a bug bounty in place. If you find a real bug, mail them and they'll pay you a bounty!

The firefox add-on website already allows donations for firefox add-ons, handled through paypal. 500-2000 dollar per day. But the marketplace will process much larger amounts of money, so they needed to increase their paranoia level.

For online payments, you need tokens and credentials. And they need to be stored somewhere. And suddenly you're a big fat juicy target just waiting to be hacked.

• XSS (cross site scripting) is an oft-occurring problem. Django has build-in protection for common cases. There's also content security policy that further limits it.

  They also started navigator.mozpay.

• Phishing. In-person tricks. For instance for getting your hand on a database for test usage. You do need something for debugging, so they now create an automatic anonymized debug database.

• SQL injection and so. They now have a REST api (solitude) for payments. This isolation helps preventing injections. Inside the database, lots is encrypted. And several items are stored outside of the databases. At the moment, the transaction data is separated from the payments data which is separated from the payment provider credentials.

  This is defence in depth: hedging against your own stupidity, basically.

  Access can happen through requests and oath1. Andy uses curling and slumber.

  There is a list of common problem points: OWASP. After reading through it they started django-paranoia which for instance provides paranoid forms: if you submit more key/values than expected, it will be logged. Also something that watches if your user agent changes during a session... IP changes are also logged, but normally they'll be valid. But if the first IP is in Poland and 5 minutes later it is in China...

About the phone: version 1 isn't done yet, but very very nearly. Which means they'll start to have scaling problems soon which need solving :-)

Tarek Ziadé can't believe he's giving a talk about his circus process manager in an actual circus tent :-)

A typical deployment is with nginx and gunicorn or uwsgi. But you add more and more items right next to your django process(es). Celery or haystack for instance. So you add a supervisor that starts 'em all. An often-used one is supervisord. You could use a system-level tool like upstart, but you need root access for that. You don't need it for supervisord.

Supervisord has some missing features like a powerful web console, clustering, realtime output, remote access and so. Supervisord has some of this, but not good enough. So they (mozilla) started with Circus.

They used several existing libraries, like psutil, zeroMQ, socket.io. psutil is the core of the system. Very handy for interacting with processes. It was a bit slow, but together with the psutils author they managed to make it fast.

ZeroMQ is an async library for message passing, so more or less a smart socket. They use message passing for making the various process data available to the circus tools, like 'circus-top' or 'circusd-stats'.

And because everyting is nicely decoupled, it is possible to add your own plugins for custom interaction. There are already community-provided plugins available.

He showed the web interface: looks nice. Live graph per process with memory and CPU usage. A simple "+" to add an additional process.

One last thing: there are multiple levels of supervision. Supervisord or circus must be started in some way by the system. And gunicorn, launched by supervisord or circus, itself starts up django processes. They added chaussette as a wsgi runner that can also run new processes on already-opened sockets so that they can be managed by circus, too.

I'm at the 2013 European djangocon in Warsaw! Ready for three days of conferencing and, for me also, live blogging :-)

Russell Keith Magee started off the conference. He remembered Malcolm Tredinnick, mentioning his code contributions, but especially his community involvement. Lots of mailinglist messages. Lots of personal involvement, too, as he visited many people and local communities. Not only Django: also chess, for instance. And he build a community here, too: working on the Australian chess community.

He passed away unexpectedly a few months ago.

Make the most of the time you have. It can be over quickly. And especially: be part of communities. Make communities work. And especially make this Django community work. Make friends. Enjoy our friendly community!

I took the train from Utrecht (NL) to Warsaw today. I only had to change in Amersfoort (NL) and Berlin (DE), so it was a pretty direct connection. 12 hours of train time (which I enjoy). So that's enough time to read through two scoops of Django, the Django book by Daniel Greenfeld and Audrey Roy! Here's my review.

The summary: buy the book and learn a lot. For the longer version, I'll simply go though my notes I made for each chapter.

Coding style

The book starts off good, in my opinion, because it tells you to write good and neat code. PEP8. Good not-too-short variable names. And it got me thinking by advocating explicit relative imports ("from .models import SomeModel"). That's what's good about this book: Daniel and Audrey state preferences and tell you best practices and sometimes those best practices won't be your best practices. Or you didn't know something. Anyway, it gets you thinking; which is good and enjoyable.

Virtualenv

Hey, virtualenv in chapter two! Nice. Here, like in the rest of the book, I noticed they point a lot at existing documentation and don't provide much explanation. No virtualenv explanation here, for instance, just a pointer to the official docs. It is not necessarily bad, probably even good, but it is something to keep in mind. You'll have to do some work yourself (which will make you retain the knowledge you gain better anyway!).

Project layout

I got tickled here. Directories three levels deep? Especially having urls and settings in a subdirectory within a directory with the very same name (the name of the project)? Two chapters later, the settings seem to be moved to a settings/ directory, so mayhap I looked at an older beta version of the book :-)

Apps

The gospel that a Django application should do one thing only is repeated here, which is a very, very good thing. I completely agree. Advice like this is what makes it a good book. You get a good mindset out of the book.

Settings

The idea to have a directory settings/ with a base.py and then production.py, dev.py, reinout_dev.py and so looks OK. I use a different setup and this proposed one looks better. Half the chapter is about enviroment variables as a means of keeping things like SECRET_KEY and database passwords out of the settings. Yep, that can work. My opinion is that you can also keep it in the settings, provided you keep your code non-public. If you use environment settings, you still need to store the data somewhere. You won't type it in by hand, will you? There's no real suggestion in the chapter to solve this, though the solution of course depends on your chosen setup (and there are too many different kinds of setups to provide a single right answer). Nice touch is the clean suggested ImproperlyConfigured error when a enviroment variable is missing, this shows the care that went into the book.

Models

Hey, I learned something new! I didn't know about auto_now and auto_now_add on DateTimeFields! That's what I read books like this for: getting hints like this.

Views

Reminder for myself that I took out of these chapters: put less code in views.py. And I ought to look at django-braces for handy Class Based View mixins.

Forms

Hm. I probably ought to use forms much more. Especially for those spots in my code where I just take two or three variables directly from the GET or POST and stuff it in some query... Why not use a small form, just for the form validation? Much safer that way and I'd use more of what Django gives me! Again something in the book that educates me :-)

REST

It took me a while to spot the difference between the two views that are shown at the start. It turns out that one is a view on the collection of items and the other a view on one single item. The first has list+add, the second view+edit+delete. The names just don't make it clear. I think this chapter is a bit too short. On the other hand, perhaps one extra paragraph and two better class names would be enough.

Templates

Flat is better than nested. Solid advice not to go overboard with blocks and template inheritance. Oh, and TEMPLATE_STRING_IF_INVALID is handier than I thought for template debugging as you can add a %s to the string, which shows you the failed expression. This tip is going to help a lot.

Admin

They say that the admin should only be used for site admins, not for end users. It is just as easy and probably better to make a couple of quick edit pages or dashboards for your customers.

Third-party apps

There are lots of apps you can use. Look at http://djangopackages.com. Did you know it was written by the same people that wrote this book? Now you know why you should read the book.

Testing

Most of my favourite/essential packages are mentioned: coverage.py, factory_boy, mock. And the tip to zap the tests.py file and replace it with a tests/ subdirectory full of test files is correct.

Documentation

Documentation is mandatory. Even when installation is done with a fabfile or with chef, tell it anyway in the installation docs. And describe what the goal of the app is. Etc. Documentation is mandatory.

Performance tuning

Debug toolbar: yes. Hey, but I didn't know yet about django-cache-panel to see what happens in the cache. Sounds handy. This chapter also whacks me on the fingers a bit as I have almost done nothing with sql/db level optimization/fixing/profiling.

Security

"Always use https". And lots of other good tips. And, for me, the reminder to use forms (or rather, form validation) more for better security.

Logging

Good that the book mentions logger.exception('Something went wrong'), as it logs the message at ERROR level and automatically includes the traceback. No more weird exc_info-like stuff, just logger.exception().

Django utils

A handy list of utils that Django already provides such as sluggify, strip_tags and so.

Deployment

Gunicorn and mod_wsgi. Personally, I'm happy with gunicorn (when run behind supervisord). Nice isolation. Nice mostly-transparent restarts when things barf.

Getting help

Good thing: they tell you to do your homework before asking for help in the usual channels. Very good.

I've got one big gripe with the book. There's probably a good reason for the omission, but I'm missing setup.py. Telling to use a certain requirements.txt in some README is a poor substitute for Python's automatic dependency handling. This is not only good for apps you want to put on PYPI, but also for your own packages.

All in all: valuable book, buy it!

Daniel and Audrey are at Djangocon in Warsaw, so if you're there say them "hi" to thank them for the book.

Today it is 150 years ago that Stonewall Jackson died. Not everyone will recognize the name: it is a general from the American civil war. And a good one at that!

Bear with me, I'll have a programming-related comment to make on documentation :-)

If you know a bit about the second world war, you might have heard about the German general Erwin Rommel. Jackson's fame was a bit like that. If you had to fight Jackson or Rommel, it didn't really matter that you had more men and equipment: he'd beat the crap out of you anyway. Once at a time Jackson's 15000 men ran circles around 60000 opponents and repeatedly beat them. That's 1:4. And they won.

Both Jackson and Rommel seemed to have a Fingerspitzengefühl. They'd known instinctively when to do or not do something. When to lay in wait and when to strike out despite the odds.

Both also seemed to be one-of-a-kind. I mean it in the sense that they could not teach others to do the same. It was all in their own head. It was all dependent upon them. And at least Jackson didn't tell anything to his subordinates; he was secretive. When he died, there was no one to take his place and no one who could emulate his expertly handling of his army.

Here's the link with programming: document your stuff. If it isn't documented, it doesn't exist. I updated a small internal tool two days ago and had to figure out which commandline arguments to pass to it because I had not originally documented it! There was a README, but only with server installation instructions; no local test instructions. And it only described one of the two scripts. Needless to say, I've now corrected this situation.

I wrote the tool originally and I'm the only one working on it. But after I haven't touched it for a year I sure need a reasonable README to get myself back on track. So: document! Try to pass on knowledge.

Btw, Stonewall Jackson died because he was shot by friendly troops. I'm not suggesting programmers should be shot for not-documenting their stuff, but a forceful reminder here or there could be useful :-)