losee losee - 4 months ago 40
Python Question

My celery redis task isn't working in my django app on heroku server

I have a task that was working fine on my local server but when I pushed it to Heroku, nothing happens. there are no error messages. I am a newbie when it comes to this and locally I would start the worker by doing

celery worker -A blog -l info.

So I'm guessing that's the issue may have to do with that. because I don't know to do this. I doubt I'm supposed to do this in my app. heres my code


import os

from celery import Celery

from django.conf import settings

# set the default Django settings module for the 'celery' program.
'DJANGO_SETTINGS_MODULE', 'gettingstarted.settings'

app = Celery('blog')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

my tasks.py

import requests
import random
import os

from bs4 import BeautifulSoup
from .celery import app
from .models import Post
from django.contrib.auth.models import User

def the_star():
def swappo():
user_one = ' "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" '
user_two = ' "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)" '
user_thr = ' "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko" '
user_for = ' "Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:10.0) Gecko/20100101 Firefox/10.0" '

agent_list = [user_one, user_two, user_thr, user_for]
a = random.choice(agent_list)
return a

headers = {
"user-agent": swappo(),
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
"accept-encoding": "gzip,deflate,sdch",
"accept-language": "en-US,en;q=0.8",

# scraping from worldstar
url_to = 'http://www.worldstarhiphop.com'
html = requests.get(url_to, headers=headers)
soup = BeautifulSoup(html.text, 'html5lib')
titles = soup.find_all('section', 'box')
name = 'World Star'

if os.getenv('_system_name') == 'OSX':
author = User.objects.get(id=2)
author = User.objects.get(id=3)

def make_soup(url):
the_comments_page = requests.get(url, headers=headers)
soupdata = BeautifulSoup(the_comments_page.text, 'html5lib')
comment = soupdata.find('div')
para = comment.find_all('p')
kids = [child.text for child in para]
blu = str(kids).strip('[]')
return blu

cleaned_titles = [title for title in titles if title.a.get('href') != 'vsubmit.php']
world_entries = [{'href': url_to + box.a.get('href'),
'src': box.img.get('src'),
'text': box.strong.a.text,
'comments': make_soup(url_to + box.a.get('href')),
'name': name,
'url': url_to + box.a.get('href'),
'embed': None,
'author': None,
'video': False
} for box in cleaned_titles][:10] # The count

for entry in world_entries:
post = Post()
post.title = entry['text']
title = post.title
if not Post.objects.filter(title=title):
post.title = entry['text']
post.name = entry['name']
post.url = entry['url']
post.body = entry['comments']
post.image_url = entry['src']
post.video_path = entry['embed']
post.author = entry['author']
post.video = entry['video']
post.status = 'draft'
post.tags.add("video", "Musica")
return world_entries

my views.py

def shopan(request):
return redirect('/')

I have multiple instances of REDIS_URL

soI ran

heroku redis:promote REDIS_URL

and that's what I use in my environment variable which you can see above. How can i make this work?


You need to add an entry in your Procfile to tell Heroku to start the Celery worker:

worker:celery worker -A blog -l info