Avoid recursive save() when using celery to update Django model fields

Avoid recursive save() when using celery to update Django model fields

I'm overriding a model's save() method to call an asynchronous task with Celery. That task also saves the model, and so I end up with a recursive situation where the Celery task gets called repeatedly. Here's the code:
Model's save method:
def save(self, *args, **kwargs):
    super(Route, self).save(*args, **kwargs)
    from .tasks import get_elevation_data
    get_elevation_data.delay(self)

get_elevation_data task:
from celery.decorators import task

@task()
def get_elevation_data(route):
    ...
    route.elevation_data = results
    route.save()

How can I avoid this recursion?

Solutions/Answers:

Answer 1:

Add a keyword argument that tells save not to recurse:

 def save(self, elevation_data=True, *args, **kwargs):
   super(Route, self).save(*args, **kwargs)
   if elevation_data:
     from .tasks import get_elevation_data
     get_elevation_data.delay(self)

And then:

 from celery.decorators import task

 @task()
 def get_elevation_data(route):
     ...
     route.elevation_data = results
     route.save(elevation_data=False)

References

Django form field label translations

Django form field label translations

I have a baseform with over 20 fields. Then I have about 15 other forms inheriting from that form, passing in a parameter called fields which the baseform uses to delete all other fields. Best explain via example:
class BaseForm(forms.Form):
    reportid = forms.HiddenInput()
    fromdate = forms.DateField(label=_("From"), widget=widgets.AdminDateWidget())
    todate = forms.DateField(label=_("To"), widget=widgets.AdminDateWidget())
    sort_by = forms.ChoiceField(label=_("Sort by"), choices=[])
    .......

    def __init__(self, *args, **kwargs):
        fields = kwargs.pop('fields')
        #Pseudo:
        ***del self.fields[field] for field not in fields***

class SubForm(forms.Form):
    def __init__(self, *args, **kwargs):
        fields = ['reportid', 'todate']
        super(SubForm, self).__init__(fields=fields, *args, **kwargs)

The resulting form would then look like this:
class SubForm(forms.Form):
    reportid = forms.HiddenInput()
    todate = forms.DateField(label=_("To"), widget=widgets.AdminDateWidget())

My problem is that when the BaseForm is initialized for the first time, the labels are bound to the fields with the active language, and when another user logs in with another language setting (or the current user changes languages) the field labels don't update.
I've come to a solution using a dict like this:  
labels = {
    'todate': lambda: _("To"),
    'fromdate': lambda: _("From"),
    .....
}

and then when initializing the baseform looping through all fields and setting 
self.fields[field].widget.label = labels[field]()

Do I have any nicer (read: more pythonic) way of achieving this?

Solutions/Answers:

Answer 1:

Django provides _lazy variants of the translation functions (for example ugettext_lazy) so you can ark strings for translations at the access time (as opposed to when the translation function is called).

It’s documented in details at https://docs.djangoproject.com/en/1.7/topics/i18n/translation/#lazy-translation

References

Adding link to django admin page

Adding link to django admin page

I have a client that has an app built with django. On every page of
their app is a link to their admin site. They tell me the admin site
is generated entirely by django, and they've never customized it
before. On the very first line of the admin page it says:

Django administration          Welcome, admin. Change password / Log out

They want me to add a link to that line, to the left of "Django
administration" that will take them back to the page they were on when
they clicked on the link to get them to the admin site.
So I have 2 issues here:

How do I override that line to add the link? It appears that page
is generated by contrib/admin/templates/admin/base.html, and I tried
to override it by following the directions at
https://docs.djangoproject.com/en/1.2/ref/contrib/admin/#overriding-a...,
but whatever I do seems to have no effect.
How can I get the link of the page of the app they came from? It's
not simply just going back one page, as they could have navigated all
over the place of the admin site before clicking the "Back to app"
link.

Solutions/Answers:

Answer 1:

There are many ways to store the last visited non-admin url in request.session. For example, a middleware:

import re

class LastSiteUrl(object):
    def is_admin_url(self, url):
        return re.search('^(http:\/\/.*){0,1}\/admin\/', url) is not None

    def process_request(self, request):
        if self.is_admin_url(request.path) and \
            not self.is_admin_url(request.META.get('HTTP_REFERER','')):
            request.session['last_site_url'] = request.META.get('HTTP_REFERER','')

Then override the template:

  1. Store the last non admin url in request.session, e.g. put the above class in yourproject/middleware.py, add to settings.MIDDLEWARE_CLASSES: middleware.LastSiteUrl

  2. Prepare the admin base site template for overriding, copy django/contrib/admin/templates/admin/base_site.html to yourproject/templates/admin/base_site.html

  3. Link to request.session.last_site_url, e.g. in yourproject/templates/admin/base_site.html, find {% block branding %}, before the H1 tag of this block, add an HTML link to {{ request.session.last_site_url }}.

It should look like that:

{% block branding %}
    {% if request.session.last_site_url %}
        <a href="{{ request.session.last_site_url }}">back to site</a>
    {% endif %}
    <h1 id="site-name">{% trans 'Django administration' %}</h1>
{% endblock %}

Answer 2:

The easy way:

If a model has a get_absolute_url function, the admin ‘change’ pages contain a ‘view on site’ button on the top right of the page. So make sure your Model contains get_absolute_url:

    def get_absolute_url(self):
        return '/myapp/%s' %self.slug #this should reflect your url-structure.

The ‘view on site’ button is not on the ‘add’ page because the object needs to be created before you can visit it. Tell your client when he/she is creating a new object to NOT hit ‘save’ but ‘save and continue editing’ and than click on ‘view on site’.

Now it’s also easy to jump to the website representation of the current model entry by overriding the base_site.html. Here we check if the current model contains absolute_url. If you are not on a change_page, there won’t be a absolute_url. In this case the link takes you to the homepage.

templates/admin/base_site.html

{% extends "admin/base.html" %}
{% load i18n %}

{% block title %}{{ title }} | {% trans 'Django site admin' %}{% endblock %}

{% block branding %}
<h1 id="site-name">{% if has_absolute_url %}<a href="../../../r/{{ content_type_id }}/{{ object_id }}/" class="viewsitelink">{% else %}<a href="http://{{ site.domain }}">{% endif %}{{ site.domain }}</a>    

 Sitebeheer{#{% trans 'Django administration' %}#}</h1>
{% endblock %}

{% block nav-global %}{% endblock %}

Done!

The following solution does also answer your question but in a different way. This will add a ‘save and view on site’ button next to ‘save and continue’ and ‘save’ buttons on the admin ‘add’ and ‘change’ pages:

  1. Make sure your model contains a get_absolute_url function.
  2. Override the admin response to redirect to this absolute_url.
  3. Override change_form.html to add a ‘save and view on site’ button.

Edit myproject/myapp/models.py

class MyModel(models.Model):
    title = models.CharField("titel", max_length=200)
    slug = models.SlugField(unique=True)

    def __unicode__(self):
        return self.title

    def get_absolute_url(self):
        return '/myapp/%s' %self.slug

Edit myproject/myapp/admin.py

from models import MyModel

class MyModelAdmin(admin.ModelAdmin):
    ...

    def response_change(self, request, obj):
        """
        Determines the HttpResponse for the change_view stage.
        """
        if request.POST.has_key("_viewsite"):
            msg = (_('The %(name)s "%(obj)s" was changed successfully.') %
                   {'name': force_unicode(obj._meta.verbose_name),
                    'obj': force_unicode(obj)})
            return HttpResponseRedirect(obj.get_absolute_url())
        return super(MyModel, self).response_change(request, obj)

admin.site.register(MyModel, MyModelAdmin)

Create a new file myproject/templates/admin/myapp/change_form.html:

{% extends "admin/change_form.html" %}
{% load i18n %}
{% block content %}
{{ block.super }}
<script type="text/javascript">//<![CDATA[
    (function($){
        $('<input type="submit" value="Save and view on site" name="_viewsite" />')
        .prependTo('div.submit-row');
    })(django.jQuery);
//]]></script>
{% endblock %}

The template will override the change_form for each model in the myapp. This could be undesirable in your situation because not all models have a representation in the website (don’t have and don’t need get_absolute_url). I think you can also put the template at myproject/templates/admin/myapp/MyModel/change_form.html to only override the template for MyModel and leave other models in myapp with de default template. I never used a change_form template for a single model. Will you let me know if it worked?

A big thank you for: http://djangosnippets.org/snippets/2005/

Did this answer your question? I hope it helps.

References

Trying to serve django static files on development server – not found

Trying to serve django static files on development server – not found

I've followed the instructions in this question, the documentation, and I've even looked at this one, but so far I'm unable to get at my static files using python manage.py runserver.
These are my (relevant?) settings:
STATIC_ROOT '/home/wayne/programming/somesite/static'
STATICFILES_DIRS ('/home/wayne/programming/somesite/static/styles',
                  '/home/wayne/programming/somesite/static/admin')

In my urls.py:
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
# The rest of my urls here...
if settings.DEBUG:
    urlpatterns += staticfiles_urlpatterns()

I have the following code in my base.html
    
    {% block styles %}
    {% for sheet in styles %}
    
    {% endfor %}

And I promise I'm not hallucinating:
(.env)wayne:~/programming/somesite/static$ pwd 
/home/wayne/programming/somesite/static 
(.env)wayne:~/programming/somesite/static$ ls 
admin  styles 

However, when navigate to http://localhost:8000/ my site is missing its stylesheets, and when I go to http://localhost:8000/static/styles/main.css I get 'styles/main.css' could not be found, and trying to navigate to localhost:8000/static/, or localhost:8000/static/styles it tells me that Directory indexes are not allowed.
What am I missing here?

Solutions/Answers:

Answer 1:

Django’s handling of static files continue to be slightly confusing, particularly in terms of the naming of relevant settings.

The short answer is to move your static files; instead of

/home/wayne/programming/somesite/static

put them in

/home/wayne/programming/somesite/yourapp/static

(where “yourapp” is obviously the name of your main application).

The longer answer is that I think you’ve (understandably) become confused about the various settings. STATIC_ROOT only refers to the location where your static files should end up after running manage.py collectstatic. You don’t need this set (as you shouldn’t really need collectstatic) when developing locally. Either way, STATIC_ROOT should always refer to an empty directory.

Your STATICFILES_DIRS setting would almost work, except that you’ve told Django there are two paths where it should find static files

/home/wayne/programming/somesite/static/styles
/home/wayne/programming/somesite/static/admin

so when you do {% static "styles/main.css" %} it will look for

/home/wayne/programming/somesite/static/styles/styles/main.css
/home/wayne/programming/somesite/static/admin/styles/main.css

and will obviously not find them. What might work is

STATICFILES_DIRS = ('/home/wayne/programming/somesite/static',)

but there’s no need to do that, as you can just rely on django.contrib.staticfiles.finders.AppDirectoriesFinder (in the default STATICFILES_FINDERS) and move your static files to an app directory.

Hope this clears things up a little.

References

Django REST Framework – POSTing foreign key field containing natural key?

Django REST Framework – POSTing foreign key field containing natural key?

I've recently started using the Django REST Framework (and Django, and Python - I'm an RTOS/embedded systems person!) in order to implement a RESTful Web API. Haven't had any problems yet which couldn't be resolved with Google, but this one has had me stumped for a few hours now.
I have an embedded system which listens for events which are associated with a range of devices - analogous to a Phone making Calls, which is what I'll discuss here for brevity. A Phone has a number and a whole lot of Calls (that it has made) associated with it. A Call has an associated Phone (the Phone which made the Call) and a time of creation. When a Call occurs, it should be POSTed to the API. I have an embedded system which listens for Calls and their originating phone number, and submits them to the API. Since the embedded system knows the phone number, I would like it to submit: {"srcPhone":12345678} rather than {"srcPhone":"http://host/phones/5"}. This avoids the need for my embedded system to know the primary key of every Phone (or to GET Phones by number every time it wants to submit a Call).
Google and the Django docs suggested I could achieve this with natural keys. My attempt follows:
models.py
from django.db import models
from datetime import datetime
from pytz import timezone
import pytz
from django.contrib.auth.models import User

# Create your models here.
def zuluTimeNow():
    return datetime.now(pytz.utc)


class PhoneManager(models.Manager):
    def get_by_natural_key(self, number):
        return self.get(number=number)


class Phone(models.Model):
   objects     = PhoneManager()
   number      = models.IntegerField(unique=True)

   #def natural_key(self):
   #    return self.number

   class Meta:
      ordering = ('number',)


class Call(models.Model):
    created    = models.DateTimeField(default=zuluTimeNow, blank=True)
    srcPhone   = models.ForeignKey('Phone', related_name='calls')

    class Meta:
        ordering = ('-created',)

views.py
# Create your views here.
from radioApiApp.models import Call, Phone
from radioApiApp.serializers import CallSerializer, PhoneSerializer
from rest_framework import generics, permissions, renderers
from rest_framework.reverse import reverse 
from rest_framework.response import Response
from rest_framework.decorators import api_view

@api_view(('GET',))
def api_root(request, format=None):
    return Response({
        'phones': reverse('phone-list', request=request, format=format),
        'calls': reverse('call-list', request=request, format=format),
    })


class CallList(generics.ListCreateAPIView):
    model = Call
    serializer_class = CallSerializer
    permission_classes = (permissions.AllowAny,)

class CallDetail(generics.RetrieveDestroyAPIView):
    model = Call
    serializer_class = CallSerializer
    permission_classes = (permissions.AllowAny,)

class PhoneList(generics.ListCreateAPIView):
   model = Phone
   serializer_class = PhoneSerializer
   permission_classes = (permissions.AllowAny,)

class PhoneDetail(generics.RetrieveDestroyAPIView):
   model = Phone
   serializer_class = PhoneSerializer
   permission_classes = (permissions.AllowAny,)

serializers.py
from django.forms import widgets
from rest_framework import serializers
from radioApiApp import models
from radioApiApp.models import Call, Phone

class CallSerializer(serializers.HyperlinkedModelSerializer):
   class Meta:
       model = Call
       fields = ('url', 'created', 'srcPhone')

class PhoneSerializer(serializers.HyperlinkedModelSerializer):
   calls = serializers.ManyHyperlinkedRelatedField(view_name='call-detail')
   class Meta:
      model = Phone
      fields = ('url', 'number', 'calls')

To test, I create a Phone with number 123456. Then I POST {"srcPhone":123456} to http://host/calls/ (which is configured in urls.py to run the CallList view). This gives an AttributeError at /calls/ - 'int' object has no attribute 'startswith'. The exception occurs in rest_framework/relations.py (line 355). Can post the entire trace if it'll be helpful. Upon reading relations.py, it looks like the REST Framework is not looking up Phones by number, but processing the srcPhone attribute as if it was a URL. This would normally be true, but I want it to look up Phones by natural key, rather than provide the URL. What have I missed here?
Thanks!

Solutions/Answers:

Answer 1:

What you’re looking for is SlugRelatedField. See docs here.

but processing the srcPhone attribute as if it was a URL.

Exactly. You’re using HyperlinkedModelSerializer, so the srcPhone key is using a hyperlink relation by default.

The 'int' object has no attribute 'startswith' exception you’re seeing is because it’s expecting a URL string, but receiving an integer. Really that ought to result in a descriptive validation error, so I’ve created a ticket for that.

If you instead use a serializer something like this:

class CallSerializer(serializers.HyperlinkedModelSerializer):
    srcPhone = serializers.SlugRelatedField(slug_field='number')

    class Meta:
        model = Call
        fields = ('url', 'created', 'srcPhone')

Then the 'srcPhone' key will instead represent the relationship using the 'number' field on the target of the relationship.

I’m planning on putting in some more work to the relationship documentation at some point soon, so hopefully this will be more obvious in the future.

Answer 2:

(Can’t post this as a comment, too long)

Tom’s answer above solved the problem.

However, I also want to have a hyperlinked field back to the Phone resource. The SlugRelatedField allows me to submit with an integer field belonging to the Phone, but when GETting the resulting Call resource, it also serialises as an integer. I’m sure this is intended functionality (doesn’t seem very elegant to have it serialising from an integer, but to a hyperlink). The solution I found was to add another field to the CallSerializer: src = serializers.HyperlinkedRelatedField(view_name='phone-detail',source='srcPhone',blank=True,read_only=True) and add that field to the Meta class. Then I POST only srcPhone (an integer) and GET srcPhone plus src, which is a hyperlink to the Phone resource.

References

Django Pipeline, Heroku, and SASS

Django Pipeline, Heroku, and SASS

I've been trying to get django-pipeline setup so that I can compile and concat my assets. I would also like to remove the compiled css files from my repository to avoid merge conflicts in pull requests.
I've been trying to get django-pipeline to compile the files as part of the deploy process but can't figure this out. I use SASS to write my CSS. My pipeline settings look like this:
STATICFILES_STORAGE = 'pipeline.storage.PipelineCachedStorage'

PIPELINE_CSS = {
    'main': {
        'source_filenames': (
            'sass/blah.scss',
            'sass/main.scss',
        ),
        'output_filename': 'css/main.css',
        'extra_context': {
            'media': 'screen',
        },
    },
}

PIPELINE_COMPILERS = (
  'pipeline.compilers.sass.SASSCompiler',
)

This works great locally, and generates .css files in my /sass folder, which are then combined to make the main.css file. If I check those CSS files into my git repository and push to Heroku, it also works fine. However, if I ignore them, which I would like to do so that I'm not committing compiled files, then django-pipeline can't find the files to combine. I'm not sure how I can get the sass compilation working on Heroku and I can't find anything about it.
I can provide more information about my setup if needed, hopefully somebody knows something about this!

Solutions/Answers:

Answer 1:

OK, here’s how I got this working using Compass to compile my SASS files.

  • Use multiple Heroku buildpacks – Heroku Buildpack Multi
  • Put the following in your .buildpacks file

    https://github.com/heroku/heroku-buildpack-ruby.git
    https://github.com/heroku/heroku-buildpack-nodejs
    https://github.com/heroku/heroku-buildpack-python.git
    
  • Create a Gemfile with compass and any other requirements you have. Here’s mine:

    source 'https://rubygems.org'
    
    ruby '1.9.3'
    
    gem 'bootstrap-sass'
    gem 'compass'
    
  • Create a config.rb file. Here’s mine. As you can see it, requires the bootstrap-sass that I included in my Gemfile:

    # Require any additional compass plugins here.
    require 'bootstrap-sass'
    
    # Set this to the root of your project when deployed:
    http_path = "/"
    css_dir = "app_folder/static/css"
    sass_dir = "app_folder/static/sass"
    images_dir = "app_folder/static/images"
    
    output_style = :compact
    

    more details on config.rb can be found here

  • Install node packages (django-pipeline wants yuglify). You’ll need a package.json file:

    {
      "dependencies": {
        "yuglify": "0.1.4"
      },
      "engines": {
        "node": "0.10.x",
        "npm": "1.3.x"
      },
      "repository": {
        "type": "git",
        "url": "your repo url"
      }
    }
    
  • almost ready…
  • when Heroku runs the ruby buildpack, it will look for a rake task called assets:precompile. So now you’ll need to add a Rakefile with the following:

    namespace 'assets' do
      desc 'Updates the stylesheets generated by Sass/Compass'
      task :precompile do
        print %x(compass compile --time)
      end
    end
    

    this will put compile your stylesheets. You need to make sure you set the output (back in config.rb) to the place that django-pipeline is looking for CSS files (shown in the original question).

  • You should get rid of this part in the original question as django-pipeline isn’t compiling your SASS for you:

    PIPELINE_COMPILERS = (
      'pipeline.compilers.sass.SASSCompiler',
    )
    
  • That should be it! Deploys should just work now, and it didn’t really add a significant amount of time to my deploy process.

All in all, it amounts to a lot of setup, but for me it was well worth it as I no longer have to commit compiled files into the repository, which was causing a lot of merge conflicts when working with branches and pull requests.

I would like to figure out how to do this using only two buildpacks (obviously only one would be ideal but I don’t know if it’s possible). The problem is trying to find binary paths so that pipeline can do it’s thing when it doesn’t find the defaults. I’m not sure if the reason I can’t do this is because of how Heroku is installing things, or because there is a bug in django-pipeline, but right now this is good enough for me.

If you try this and it doesn’t work for you, let me know, if I missed something I’m happy to make updates.

Answer 2:

I don’t want to take away from your excellent solution, but I tried this today and found a few differences that made things simpler for me – likely due to updates in django-pipeline and/or Heroku. My full solution is below, in case anyone else comes looking.

Add the 3 buildpacks to Heroku:

heroku buildpacks:set https://github.com/heroku/heroku-buildpack-ruby.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-nodejs
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-python.git

Add django-pipeline and django-pipeline-compass to requirements.txt:

django-pipeline==1.5.2
django-pipeline-compass==0.1.5

Create a Gemfile to install Sass:

source 'https://rubygems.org'
ruby '2.1.5'
gem 'bootstrap-sass'

Create a package.json file to install Yuglify:

{
  "dependencies": {
    "yuglify": "0.1.4"
  },
  "engines": {
    "node": "0.10.x",
    "npm": "1.4.x"
  }
}

I did not need a Rakefile or config.rb.

For reference, here are relevant settings from my settings.py:

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, '_generated_media')
STATICFILES_STORAGE = 'pipeline.storage.PipelineCachedStorage'

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'pipeline.finders.PipelineFinder',
)

PIPELINE_COMPILERS = (
    'pipeline_compass.compiler.CompassCompiler',
)

PIPELINE_YUGLIFY_BINARY = os.path.join(BASE_DIR, 'node_modules', '.bin', 'yuglify')

And I also had to add this entry to urls.py:

url(r'^static/(?P<path>.*)$', serve, kwargs={'document_root': settings.STATIC_ROOT})

Hope it helps someone!

Answer 3:

You may need to set PIPELINE_SASS_BINARY so that django-pipeline can find your SASS compiler.

Answer 4:

You can use the libsass compiler for django-pipeline that uses Sass packaged as a Python package:

pip install libsasscompiler

Update your config:

PIPELINE['COMPILERS'] = (
  'libsasscompiler.LibSassCompiler',
)

The default Yuglify compressor is also a problem on Heroku, which you can temporarily overcome by disabling it. This is my config for enabling Sass on Django for example:

PIPELINE = {
    'COMPILERS': (
        'libsasscompiler.LibSassCompiler',
    ),
    'STYLESHEETS': {
        'main': {
            'source_filenames': (
              'styles/main.scss',
            ),
            'output_filename': 'css/main.css'
        },
    },
    # disable the default Yuglify compressor not available on Heroku
    'CSS_COMPRESSOR': 'pipeline.compressors.NoopCompressor',
    'JS_COMPRESSOR': 'pipeline.compressors.NoopCompressor'
}

The longer-term solution would be to move towards a JS-only build toolchain (as most projects are doing). Rails integrates pretty nicely with Webpack for example and is maintained by the same team. Until that happens in Django (if ever) and trickles into the Heroku Python builpack, you can use Heroku’s multiple buildpacks and add an official Node buildpack step that runs npm install; npm run build for you.

References

crispy-forms: add css class for one of the inputs

crispy-forms: add css class for one of the inputs

In my forms.py I have
[...]
self.helper.layout = Layout(
    Field('name'),
    Field('description'),
)
self.helper.form_class = 'form-horizontal'
self.helper.label_class = 'col-md-2 col-xs-3'
self.helper.field_class = 'col-md-10 col-xs-9'
[...]

which renders to
Now I would like just the name-input to be smaller, like this: [...]