Chris vCB Chris vCB - 2 years ago 87
Python Question

Passing an input to a service and saving the result to DB in Django

I have a Django application that involves a model with five fields. In the case of one of these fields, I want users to enter a bunch of text, which I then want to submit to a service (via a function call) and save the result. To provide a visual representation:

enter image description here

How does one best play this? One option to me would be to override the save() function, but the types are different - I want the form to show a models.TextField field but save it as the results of a URLField would be saved. Equally, when displayed, I want the user to edit not the URL but rather the text retrieved from that URL.

Answer Source

I think there is no easy standard way to solve your problem. (This is why I don't provide any code, treat this as a long comment instead of a solution. The update provides sources for one of the solutions discussed in this answer.) There are only solutions with pros and cons depending on your circumstances.


  1. Asynchronous processing: You are accessing an external service and the request may take some time to complete. For this reason this operation should be done in an asynchronous way (both storing and retrieving data from the external service). The problem with this is that django wasn't really invented for async tasks, there are only hacky async solutions for django.

    • By accessing the external service directly in a non-async way from your django backend your site may hang up for long periods when the external service fails (in worst case with long timeout settings at different points between your server and the external servers). For a not too serious site we can assume that the external service will work in most of the cases and if not, then a little bit of downtime for our servers is acceptable.

    • Doing async processing with a django backend is not only hacky and messy but a really good solution is sometimes nearly impossible.

      On really good solution I mean something that looks like a solution you write with tools/frameworks that have been invented for async servers (golang, gevent). Typical async solutions with django often involve very complex architecture and code compared to a solution with pure golang or gevent server code.

      For example if you have to serve the client some data that you have to retrieve from an external service then if response comes with high latency from the external service then your django backend will have to wait for the response anyway. If you do the waiting in an "asnyc-helper" server (celery, twisted, or gevent) then you may still have to write messy django response handling code with polling, long polling or websockets. The resulting code is super bloated and messy compared to pure async server code.

  2. Server-side validation of the text behind the URL, hiding th URL from the client, etc... You may or may not need these and they can influence which solution you choose.

At Which layer should you put the meat of your solution?

Possible choices:

  1. Client layer
  2. View layer
  3. Form layer
  4. Model layer

1. Client layer

If it isn't a problem that your client knows about the external service and directly communicates with it then this is probably the simplest and solution. Implementing async access to the external service is straightforward in javascript.

2. View layer

I think putting the management of this whole thing to the view layer isn't a good idea. People tend to stuff everything to the view for some reason (perhaps because it seems to be the "easiest point of attack" in case of adding new code). In my opinion the view shouldn't do much more than parsing the input from the request for the business logic (and validating it with/without forms) and then formatting the response for the client.

3/4. Form/Model layer

When you implement validation of client requests the form layer is usually the first place to consider. This is especially true if the data sent by the client has a bit different typing/layout than the underlying db models.

  • It is better to put validation to forms than into the view because forms are more reusable. Well, you can provide your own reusable validator utility functions/base-classes but still, forms have been provided exactly for this purpose as a specialized standard solution. With forms people implement validation in a more or less standard and controlled way you would expect while in case of putting custom logic into the views and validator utility functions/base classes, people generate a lot of messy code for you to read and these code snippets will always look different. Forms can often be written in a declarative way that lowers the chances of introducing bugs and makes reading code easier because you know what to expect.
  • Besides the specialized and declarative nature of forms there is another reason why is it a better layer for validation than the model layer: forms is a higher level layer and validation is also often a high level operation. It may seem a good idea to put fancy validation and tricky logic into your db models' save and pre_save signal until you have to access and/or fix the raw DB directly for some reason. It can also cause a lot of undesired conflicts between your features using models. For practical reasons it is good to have a low level layer that can be accessed without triggering fancy higher level logic.

Your solution can be implemented either as a custom form field, or a custom model field (that provides its own less-meaty form field). Implementing the form field only solution is much easier. Writing a custom model field is a pain in the ass. I've done it once and I didn't like the journey. There is a "Writing custom model fields" page in the django documentation and even that doc states that writing them isn't straightforward and recommends reading the source code of existing fields for inspiration.

In case of the model field solution I think the pastebin model field would be very similar to the standard FileField. Both of these fields basically store the value in an external storage and they store only an id to the database. The pastebin model field too should provide its own form field and widget that behaves differently than that of FileField.

At this point I didn't think enough about the pros and contras of the form field only and model field solutions but I think the model field solution can be too heavy weight with much more difficult implementation and a lot of room for errors. In case of a field that you don't want to reuse too often I think it's a bad idea to go for the model field variant. The form field is much easier and less risky.


We have discussed 3 possible solutions:

  1. Client layer implementation: The backend provides a URLField and the client uses the url to directly communicate with the external service.
  2. Form layer implementation: Using a custom form field that provides a Textarea widget and converts back-and-forth between the URLField of the model and the text area of the frontend by accessing the external service.
  3. Model layer implementation: Providing a FileField-like model field or customizing FileField itself.

The first one is quite straightforward to implement for someone with some javascript knowledge. From the second and third solutions I think the third one is more complex so let's see an example implementation of it. Key points:

  • I've tried to subclass FileField in a minimally invasive way to make the solution less fragile. The subclassing changes the default storage and the default form field associated with FileField (and adds an optional caching optimization without which the solution would still work).
  • The implementation contains a pastebin-specialized storage for our specialized FileField.
  • The custom form field converts between the text of a textarea and the contents of a pastebin file of our model field.
  • I've tried this code with success using django 1.8 and python 3.5.1 and I don't plan supporting any versions, this code is here just to provide a guideline. Use it at your own risk. If it cuts down your limbs, it is your problem. :-D



import io
import requests
from django.core.files import File
from import Storage
from django.conf import settings
from django.utils.deconstruct import deconstructible

class PastebinStorage(Storage):
    def __init__(self, options=None):
        """ The options parameter should be a dict of pastebin parameters for
        the 'create new paste' operation: see
        The most important required option is api_dev_key. Optionally you can
        set api_user_key you want to create non-guest pastes. """
        self.__options = getattr(settings, 'PASTEBIN_STORAGE_OPTIONS', {})
        if options is not None:

    def options(self):
        if 'api_dev_key' not in self.__options:
            raise ValueError('The "api_dev_key" option is missing')
        return self.__options

    def _save(self, name, content):
        # TODO: allow overriding the options on a per-file basis. Maybe we
        # should encode options into the name since we don't use it and
        # we return a completely new name/id at the end of this method.
        data = self.options.copy()
        response ='', data=data)
        # A successful response contains something like:<PASTE_KEY>
        return response.text[response.text.rfind('/')+1:]

    def _open(self, name, mode='rb'):
        if mode != 'rb':
            raise ValueError('Currently the only supported mode is "rb"')

        if 'api_user_key' in self.options:
            content = self._get_user_paste(name)
            content = self._get_public_paste(name)

        mem_stream = io.StringIO(content) = name
        mem_stream.mode = mode
        return File(mem_stream)

    def _get_user_paste(self, name):
        response ='', data=dict(
        # FIXME: Unfortunately the API seems to return status_code 200
        # also in case of errors with messages like "Bad API request,
        # invalid permission to view this paste or invalid api_paste_key"
        # in the body.
        return response.text

    def _get_public_paste(self, name):
        response = requests.get('' + name)
        return response.text

    def get_valid_name(self, name):
        return name

    def get_available_name(self, name, max_length=None):
        return name


from django.db.models import FileField
from django.db.models.fields.files import FieldFile

from .storage import PastebinStorage
from .form_field import PastebinFormField

default_storage = PastebinStorage()

# This custom FieldFile implementation is an optional optimization.
class PastebinContentCachingFieldFile(FieldFile):
    def cached_get_pastebin_content(self):
        cached = getattr(self, '_cached_pastebin_content', None)
        if cached and cached[0] ==
            return cached[1]
        with as f:
            content =
        setattr(self, '_cached_pastebin_content', (, content))
        return content

class PastebinModelField(FileField):
    attr_class = PastebinContentCachingFieldFile

    def __init__(self, verbose_name=None, name=None, storage=None, **kwargs):
        storage = storage or default_storage
        super(PastebinModelField, self).__init__(verbose_name, name, storage=storage, **kwargs)

    def formfield(self, **kwargs):
        defaults = {'form_class': PastebinFormField}
        return super(PastebinModelField, self).formfield(**defaults)


import io

from django.core.files import File
from django.forms import Textarea, CharField

class PastebinFormField(CharField):
    widget = Textarea

    def prepare_value(self, value):
        if value is None:
            return None
        if isinstance(value, str):
            return value
        # value is expected to be a PastebinContentCachingFieldFile instance
        return value.cached_get_pastebin_content()

    def to_python(self, data):
        data = super(PastebinFormField, self).to_python(data)
        if data is not None:
            mem_stream = io.StringIO(data)
   = 'unused'
            mem_stream.mode = 'rb'
            data = File(mem_stream)
        return data


The storage uses the requests library: pip install requests.

Optionally provide some default pastebin storage settings in the central django settings file. A minimal example can be something like:

    'api_dev_key' : '<your_api_dev_key>',

If you provide the pastebin config in the central config then using the pastebin FileField in your models is as simple as:

class MyModel(models.Model):
    file = PastebinModelField()

If you don't specify a central config or if you want to override the central config then you can do that per-field:

class MyModel(models.Model):
    file = PastebinModelField(storage=PastebinStorage(options=dict(

ModelForm will automatically generate a textarea with the contents of the pastebin file for the above model field. POSTing and saving with ModelForm creates a new pastebin file.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download