All posts tagged django


Last time we looked at how to generate a very simple PDF using ReportLab and Django, ReportLab and Django – Part 1 – The Set Up and a Basic Example.  This time let’s make the PDF a little bit more interesting with some headers and footers.  ReportLab gives a pretty good amount of control when it comes to adding headers and footers to your PDF.  To start off let’s bring back the simple __init__ method we had from last time and add a header/footer method to it.

Now we have a method that will draw things onto every page of our header and footer.  Next we need to actually call that method.  Remember last time where we built the document using and ReportLab built a PDF using the flowable elements in the list?  Well this time we are just going to add some more parameters to let the build process know that it needs to also draw out the header and footer on every page.  Below is our print_users method from last time with the new parameters added to the build command shown in bold.

That’s it. Now our print_users method will print out a few lines on both the header and the footer of every page. You’ll notice there are two parameters, onFirstPage and onLaterPages. This allows you to have different headers and footers on the front page, which could be a title page, and on later pages that would require page numbers. Speaking of page numbers let’s get into that.

Page numbers on every page was a big challenge for me to find anything on. It was easy enough to put Page X on every page but what I really wanted was Page X of Y and that “of Y” portion was tricky. Luckily, I’ve figured it out and I will present it here for you now.

You can alter the way that the PDF is built using the canvasmaker parameter in the process. These canvasmakers can help you inject new things on every page without using a header or footer. Below is the code that I found somewhere online to print the Page X of Y on every page down near the footer. I have placed this all within the file I created last time.

To use this we just add a new parameter to our command. You can see it added below in bold.

There you have it. Now you have a great header and footer on every page, or every page after the first, and page numbers on all of them. How fantastic is that? Next time we will go a bit deeper into Tables and how to use them to line everything up really nicely. They will be usable wherever you want a flowable including your new headers and footers.


As we continue to push forward on development our users have asked for a robust way of dragging and dropping images and linking them to a work order.  I was putting this off for a while because I thought it would be a pretty difficult task but after a few Google searches I found a great jQuery plugin from blueimp called jQuery-File-Upload.  Take a look at their demo and check out how awesome it is.  We didn’t need any of the fancy bells and whistles that came with the Basic Plus UI so this tutorial focuses on just the Basic but you can scale yours how you’d like.  The tutorial will also focus on getting it to work and save to a MySQL database through Django.

To start off let’s get the drag and drop area set up on our page.  To do that you’ll need to add these things to your HTML template, Django tags are added because that is what we are using to render the page.


With the code above you should be ready to drag and drop files immediately, at least from the front end perspective. Now let’s set up the server side of things. Make sure you add your URL from the fileupload to your file. Then create a view for that URL that looks something like this.

The view handles all of the uploaded files one at a time and saves them to wherever we would like. In this case we are making a database entry that houses the file in an ImageField and we have a CharField for a name that we can change without altering the file path used in the ImageField. The JSON output is important as it will read the data sent back and can output our newly added images right to the screen.  That’s all there is to it.

Now there may be some circumstances where you would like to add more data to the request than just the images you have dragged over.  In our case we wanted the work order number to be attached as well and saved in the file name.  Using jQuery File Upload we can add data anytime something is dropped by adding a bind call to all fileuploadsubmit method calls.  I added the following lines of code in the HTML template just underneath the var url line.

This will add the workorder id to the POST request data on every call using the fileupload. Another thing to note is you can add a check to see if you really want to submit the uploads like I have done. If the workorder is not active do not upload any images. You can customize this any way you’d like since it is all just JavaScript and jQuery code.

Another problem I ran into was having multiple file upload elements on the same page. This was really frustrating and reading through their documentation they had a fix that worked pretty well with a couple of tweaks.

First, to make all of the drag and drop areas work independently of one another you change the fileupload elements from having an id of fileupload to a class of fileupload. This will need to change on the above code for bind as well. But you can’t simply change it on the $(‘#fileupload’).fileupload() becuase then it will upload once for every drag and drop element on the page. You instead have to take it a step back and add these lines.

This allows every fileupload classed element to take files individually. But this was annoying because users had to drop the files right on the button to make it work. As you can see there is a new parameter we’ve added called dropZone. This tells the fileupload what area of the page to watch for. If you want to make this area bigger or smaller that is totally up to you. I changed mine to dropZone: $(this).parents(".panel-body") to allow the user to drop images anywhere inside the panel body and they will start the upload.

I hope this was useful to someone. We are actually going in a different direction with our image uploads but we may need this in the future and I wanted to document how I got it all working with both jQuery File Upload and Django.


Thanks to Tuan Nguyen for pointing out that in its default configuration each image is uploaded in its own request.  To join these into one request, open up the jQuery-File-Upload/js/jquery.fileupload.js file and search for the line singleUploadFiles: and set it to true.  You can also limit the number of files to be uploaded right below by changing limitMultiFileUploads:.

It’s been far too long since my last post.  A lot has happened and the holidays make things so busy but I’m back and ready to get back in to the swing of things.  To start off the new year I’m going to do a series of posts relating to ReportLab specifically with its application to Django.  I want to go over a few of the pitfalls I’ve experienced and how to get things set up.  ReportLab is notoriously tricky to get working and the documentation seems sparse to say the least.  I hope these posts help not only myself remeber how I did all of this but others who are struggling with the same problems.


The problem is simple, how to generate PDFs using Django.  In our on going rewrite of our system at work we are getting to PDF generation.  The old system uses wkhtmltopdf which is great if you just want something to render HTML out to a PDF and you don’t give much care to speed or formatting.  Wkhtmltopdf suffers from having to use HTML to format everything on the page.  This is limiting because it is difficult to add headers and footers, positioning objects on the page is tricky, and we have a bad problem with it printing blank pages due to random overflow of the elements.  We needed a better solution and we have found it with ReportLab.  ReportLab has some HTML support, and we’ll get to that in a later segment, but for the most part you are drawing elements, text, images, etc. directly onto a canvas that is then rendered as a PDF.  The speed increase for us was unbelievable.  The formatting and control that we get is unmatched.  Most importantly, the font support has given us cleaner paperwork than we ever thought possible.  I guess what I’m trying to say is ReportLab is great…once you get it to work.  ReportLab Documentation is broad but not specific enough for a lot of our concerns.  You’ll find yourself heading to Google for even the smallest of questions.  I hope to alleviate many of the common pitfalls with this series.

Getting Set Up

Start by installing reportlab into your virtual environment with pip install reportlab. The first thing to do is get a robust class set up for doing all of your printing.  I created a file that houses all of my basic ReportLab set up and functions.  Branching off from there I have my individual pages and paperwork that get combined and output to the user.  Let’s start with a simple file.

As you can see my class is very simple. I pass in a buffer to house my data that will be returned to the user, I’ll show this once we get set up, and a page size. ReportLab has a lot of built in page sizes but you can even set your own if you’d like by using your own width and height variables.

Simple Example – Printing Users

Now let’s add a simple method to print out all of the users of our application into a PDF and return it to the browser.  Inside of the MyPrint class I’m adding a method that will print the list of users.

The code is commented nicely but let’s walk through it.  We start by taking our buffer and putting it into what is a SimpleDocTemplate.  This is a ReportLab object that will generate a PDF with some standard values.  There are a ton of customizations you can make and even create your own document templates but for now let’s stick with the simple.  We set the margins for the page and the page size.  You can use inches or millimeters too if you’d like, ReportLab has a few built in measurement sizes that just need to be imported.

Next we have a list of elements.  This list will hold all of the ReportLab Flowables that will be generated on the PDF itself.  A Flowable is an object that has space and size on a PDF.  This can be a block of text, a table, an image, or you can even create a custom Flowable.  Whatever we put in this list will be generated in order on our PDF.

After that you’ll see some styles that we use.  These styles are going to be used for various elements like Paragraph styling.  After we get the style sheets I’ve added a new one that simply centers text and nothing more.

Now we come to the part where we are going to add elements to our document to be rendered.  I’ve used Django to grab my list of users and I loop through them one at a time to add their name to a Paragraph flowable.  A Paragraph flowable will take up the entire width of the area you give it.  In our case we have not confied the Paragraph to the inside of a table or to a column so each Paragraph will fill in the entire width of the document.

Once we have added all of our elements we simply build the doc.  This renders the PDF and places all PDF data into that buffer we gave the doc when we initiated it.  Close the buffer for good practice and return the contents of the PDF to the user.  Simple enough.  To run all of the code that I have shown I have a view in my file that will run all of this and you can see how to access the class and method and output your PDF to your users.

That’s all there is to it. I hope this was enlightening on at least how to get started. Next time I will explain how to add headers and footers to your documents as well as a surprisingly tricky to find way of adding page numbers.

If you have anything you’d like to learn about for ReportLab and Django please leave a comment below.

ReportLab and Django – Part 2 – Headers and Footers with Page Numbers


A couple of weeks ago it came out that there is a flaw in Django’s ImageField which could potentially allow for phising programs to be uploaded and grab cookies or do other malicious things.  While there will be no fix in Django directly you still need to take precautions on how you serve and receive files uploaded by your users.

Django has a page dedicated to fixing this exact issue.  Head over to their security guide and read up on the fixes.  They shouldn’t be too hard and shouldn’t affect any user experience.

Just wanted to post and let everyone know of the vulnerability.

Adding Custom Django 404 and 500 Pages


Django comes with a default 404 and 500 error pages which are dull and boring.  It would be much better if you add your own spin on those pages to inform your user of what happened.  Django makes this super easy by searching your project template files for a 404.html or 500.html file when it tries to render those pages.  This means all you need to do is make a nice Django template called 404.html and 500.html and place it in your project templates folder and you’re done!  You’ll have access to whatever context processors you are running for information, most of the time these pages won’t be user specific and will just display some error.

Please note that you won’t be able to see them unless you turn DEBUG = False in your file.  Once debug is off Django will render the 404 and 500 pages rather than the debug output.

Hope this helps people get their pages a little more user friendly and comfortable.

Django finally released version 1.6 of their Python based web-framework.  I use it at my job everyday and we ran into issues with the old system where they were using such an old version of Django that it was impossible to just simply update.  One of the goals with the new system we are creating is to stay as up to date as we possibly can with all of our technologies.  For us that is Bootstrap, jQuery, Django, and a few other third-party web plugins.  Here are a few of the notes I’m writing down for myself and for others to use if they run into issues in upgrading from 1.5.4 to 1.6.

Default session serialization switched to JSON

Django changed the way the session variable is being stored from pickle to JSON.  This is for the best as there is an exploit if the attacker knows your SECRET_KEY and has access to the CSRF tokens being sent by users and the session variable uses pickle.  Using JSON fixes this and Django has elected to make it the new default.  The issue is that some things, in our case timezones, are not JSON serializable.  If you’d like to still use pickle for your session variables just add SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer' to your file.

Simplejson Deprecated

Simplejson has been deprecated and you should now use JSON.  This means you’ll need to change your imports from from django.utils import simplejson to just import json and you’ll need to change any code that uses simplejson to just json.

User Profile Deprecated

They have officially deprecated the use of user profiles attached to the default user in favor of custom user models.  Custom user models should be setup on initial project creation.  If you didn’t do this then you can use something like South to migrate your data.  The easiest way to create a custom user model is shown below by extending the base Django user and just adding to it.

Now anywhere you would make a foreign key to User you’ll need to import settings and use settings.AUTH_USER_MODEL instead of User.

Those are all of the major things that I needed to change to update everything.  Granted they started to deprecate user profiles back in 1.5 we just hadn’t made the change yet and needed to now.  Hope this helps and if you find any other bugs or awkwardness when upgrading post in the comments below.  You can also read the Django 1.6 release notes on their site.

Django works wonderfully as a web framework but sufferes from sub-par data migration.  Luckily for all of us one of their developers has created an application called South that fixes this very problem.  South is meant to bring data migrations to Django and make it easy to make schema changes that are quickly applied to your database without much effort on your part.  Django does have a few built in features for dropping tables, creating tables, and will give you the code to enter to create the tables but it doesn’t help much when you are working on a production server that you don’t want to simply drop and readd tables all the time.

To get started just go through the installation tutorial.  Once you’ve done that you’re ready to migrate your data!  This is best done from the very creation of the project so South can help you from the beginning.  If you are starting a new project go here and follow along.  I’m going to focus a bit more on converting an already created project and making updates and changes to your projects.

Now that South is installed on your already created project you’ve been using for a while you’ll need to convert your app so South can interact with it.  In your console just enter the commands:

Done!  Now your app is ready to go with South.  South keeps track of the migrations using a few tables in your database which is why you need to syncdb before converting any of your apps.  After you have converted your apps you’ll notice you have some new .py files that will need to be uploaded to your repositories.  When South tries to do a migration it makes sure that you have all of the .py files that it has listed in its database for that app.  After the initial conversion, and you’ve uploaded the files, all other machines with the installation will need to run the command ./ migrate myapp 0001 –fake to ensure everyone starts on the same page.  The –fake command at the end tells South to take note of what is happening but don’t actually do it because it’s already been done.

When you make changes to your models, from now on, you’ll need to use South to do the syncdb instead of Django.  Let’s say you add a few columns to a table, remove some others, rename even more, and add a bunch of new tables.  To have South take care of all of that just use the commands:

After running the first command South will try to document all of the changes, display them to the screen, and if all goes well it will tell you to run the second. When running the second command it will actually apply the changes to the database. That’s all there is to it. Occasionally, something will go wrong. If a problem occurs at the first command it’s most likely because you have a problem in your code that is making it fail to compile. If it fails in the second command there are more issues. You could have a partial migration which they will give you code to try and clean up but I’ve found it’s best to do it yourself as their solution is usually to drop a lot of tables. It’s kind of a headache when that happens and has turned off a few of us from South but for the most part we have had a lot of success with it. If you do make all of the changes by hand just be sure and tack on the –fake at the end of the second command to let South know that the changes were already made and that they shouldn’t try and do them again. Be sure and always commit your .py South files because even if I am in a different app trying to use South I will need all of the South .py files from all of the apps before the sync can happen.

Hope this helps a few people and explains a few things about the nuances of South. It really is a great tool and they are even trying to work it into the default installation of Django in the near future so that will be exciting. If you have any tips or tricks when using South post them in the comments below.

One of the biggest things I’ve run into with Bootstrap are their tables which are dynamic and grow and shrink depending on content size.  For the most part this is completely desired.  Every now and then you want a bit more control over your table column widths.  This is a very simple trick that I wanted to document for those who have run into the problem of wanting more control than Bootstrap gives initially.

Say you’ve created your table as shown below.

Let’s say once you render this table the description for the test is way too big and is pushing the other columns away and making them smaller than you think they should be.  The quickest way to fix this is to simply give column sizes to your table headers.  Whatever the width of your table headers your table data will conform to that size.  Knowing this we can alter the above example to make description take up less space and give a little bit more to other columns.

Super simple.  Works great.  Give it a try.  The biggest request I’ve had, that I have yet to find an answer for, is how to stop the overflow for a table data and show ellipsis instead using Bootstrap.  It’s a simple enough procedure in HTML but Bootstrap is doing something else that doesn’t allow the overflow.  If anyone knows how to keep a table data on one line and have it show ellipsis please leave a comment.

A big request we had in designing our new system was an easier way for users to get data from the database.  With the large amounts of data that we use across dozens of tables it can be slow going to use the Django methods.  Instead they have given the ability to use and write raw SQL and use that result how you please.  You can read about using cursors here, I may do a post in the future about them but today I want to focus on the SQL we wrote and the problems we encountered.

The requested query was to get a list of customers, how much business we did with them, and break that up by year.  The year could be inputted by the user so it needed to be dynamic enough to scale.  The first way we tried was to create a temporary table for every year needed.  Each table was created from a SELECT statement that was specified for the office as well as the year.  Below is the SQL command for one of the offices and years but you can imagine we copied this code, almost verbatim, but changed the SELECT statement to get the correct data.

Once all of these tables were created we had to do a ton of LEFT JOINs against the main office to get the split of the total between the other offices.  Needless to say this ran very slowly.  Using Python to change the %s to a year a typical loop through creating the tables, gathering the data, and returning took roughly 16 seconds.  Yeah, not good.  The good part was that it automatically put everything into one line for each client and one column for every office total and annual total.

Obviously we had to cut down on the time.  By changing how we did our whole process we can cut the time down significantly without using JOINs.  Now this may not work in perfectly raw SQL because we are saving the output to a variable and then combining them.  Below is the new SQL code again changing the first one for each office.  The difference here is that instead of doing a table and SELECT for every year we are doing one SELECT for all the years and creating a row for each annual total.  So requesting data 5 years back could give a possible five rows for each customer, one for every year.

After every select we save the output to a variable named for the office.  In Python these are saved as lists of dictionaries.  In the end we combine the lists and then create an output of the combined totals.  I wrote an algorithm to loop through the giant list and adding the data to an output dictionary that had only one entry for each client.

By changing the way that we create the tables we were able to cut that time to ~5 seconds.  It’s a lot of data so it takes some time but 1/3 the last time is a big improvement. Obviously this code won’t work in just SQL but it is useful for Django cursors.  If there is a way to accomplish what we are doing, in a very fast query I would totally be up for hearing about it in the comments.  I have altered our actual MySQL statements to be more readable.  In the end we just wanted a list of clients and the total work orders done for a year as well as the total for each office for that year.  Our solution works and we are fairly happy with it.  Always room for improvement though.

Recently I tried sending a list through a form using something like <input name="my_list[]" value="some val">.  When this form is submitted all of the inputs with that name are grouped together for easy access.  In Django, one would think you could simply access the list using request.POST['my_list[]'].  This is not the case. Using that command will give you the last value.  Django says this is a feature and to get the list use request.POST.getlist('my_list[]').

Just an FYI and reminder to myself.