Blog

Video support

December 30, 2023 22:53 +0000  |  Blogger Python 0 

71 files changed, 2242 insertions(+), 1405 deletions(-)

It's been a very long time since I started working on supporting video in my former image gallery, but it's finally finished. This site has had a substantial overhaul, dropping the old easy-thumbnails library in favour of rolling my own thumbnailer that stores the thumbnail locations on the Media object. I also employed some light-touch polymorphism to support rendering out a page of media to include both images and video. There were a bunch of backflips required (tinkering with ffmpeg) to extract metadata from videos as well as to thumbnail them, and the geometry I had to fiddle with to make it look just right wasn't fun either.

I also dropped the old js Packery library in favour of CSS grid and then spent literally weeks testing migrations since we're talking about around 77GB of images and video that I not only didn't want to lose, but I also wanted to interrogate further for higher quality metadata. Finally, my 16-core, 64GB desktop machine was getting taxed to its limits.

I don't know how well this is going to perform on the Raspberry Pi Kubernetes cluster though. Simple image thumbnailing works just fine, but video transcoding on arm64? It's going to be interesting.

Anyway, it was a shittone of work, so I thought it worth posting about. Chances are, you won't see any of the videos unless you login since it's all of my kid anyway :-) If none of the above makes sense to you, don't feel bad. This is a very nerdy subject.

django-debreach + DRF = sadness

June 23, 2017 16:12 +0000  |  Django Python 0

I sunk 4 hours of my life into this problem yesterday so I thought I might post it here for future frustrated nerds like myself.

If you're using django-debreach and Django REST Framework, you're going to run into all kinds of headaches regarding CSRF. DRF will complain with CSRF Failed: CSRF token missing or incorrect. and if you're like me, you'll be pretty confused since I knew there was nothing wrong with the request. My token was being sent, but it appeared longer than it should be.

So here's what was happening and how I fixed it. Hopefully it'll be useful to others.

Django-debreach encrypts the csrf token, which is normally just fine because it does so as part of the chain of middleware layers in every request. However, DRF doesn't respect the csrf portion of that chain. Instead it sets csrf_exempt() on all of its views and then relies on SessionAuthentication to explicitly call CSRFCheck().process_view(). Normally this is ok, but with a not-yet-decrypted csrf token, this process will always fail.

So to fix it all, I had to implement my own authentication class and use that in all of my views. Basically all this does is override SessionAuthentication's enforce_csrf() to first decrypt the token:

class DebreachedSessionAuthentication(SessionAuthentication):

    def enforce_csrf(self, request):

        faux_req = {"POST": request.POST}

        CSRFCryptMiddleware().process_view(faux_req, None, (), {})
        request.POST["csrfmiddlewaretoken"] = faux_req["csrfmiddlewaretoken"]

        SessionAuthentication.enforce_csrf(self, request)

Of course, none of this is necessary if you're running Django 1.10+ and already have Breach attack protection, but if you're stuck on 1.8 (as we are for now) this is the best solution I could find.

Django Migrations Doesn't Play Nice with Third-Party Stuffs

September 17, 2015 18:42 +0000  |  Django Python 0

I ran into something annoying while working on my Tweetpile project the other day and it just happened to me today on Atlas. Sometimes, removing code can cause explosions with migrations -- even when they've already been run.

Example:

  • You've created a new class called MyClass.
  • It subclasses models.Model

  • It makes use of a handy mixin you wrote called MyMixin:

    class MyClass(MyMixin, models.Model):
    # stuff here

  • You create a migration for it, run it, commit your code and congratulate yourself on code well done.

  • Months later you come back and realise that the use of MyMixin was a terrible mistake, so you remove it.
  • Now migrations don't work anymore.

Here's what happened:

Creating a migration that's dependent on non-Django-core stuff to assemble the model (think mixins that add fields, or the use of custom fields etc.) means that migrations has to import those modules to run. This is a problem because every time you run manage.py migrate it loads all migration files into memory, and if those files are importing now-non-existent modules, everything breaks.

Solution:

It's an ugly one, but so far it's the only option I can figure: manually collapsing the migration stack. Basically you make sure you've run all of the migrations to date, then delete the offending classes, delete all of the migration files, and recreate a new empty migration:

$ cd /project/root/
$ ./manage.py migrate
$ rm -rf myapp/migrations/*
$ touch myapp/migrations/__init__.py
[ modify your code to remove the offending fields/mixins ]
$ ./manage makemigrations myapp

Now run this in your database:

DELETE FROM django_migrations WHERE app = 'myapp' AND name <> '0001_initial';
UPDATE django_migrations SET applied = NOW() where app = 'myapp';

The new single migration created won't be importing the removed classes, so everything will be ok, and you have the added benefit of not having so many migrations to import. Note however that this may cause problems with migrations from other apps that may have been created dependent on your now-deleted migrations, so this may start you down a rabbit-hole if you're unlucky.

I hope this helps someone in the future should this sort of thing present itself again.

New Site! (Well, Most of One Anyway)

October 04, 2010 01:41 +0000  |  Blogger Django Python Software 8

I haz a new site! I've been hacking at this for a few months now in my free time and it's finally in a position where I can replace the old one. Some of the features of the old site aren't here though, in fact this one is rather limited by comparison (no search, no snapshots, etc.) but the underlying code is the usual cleaner, better, faster, more extendable etc. so the site will grow beyond the old one eventually.

So, fun facts about this new version:

  • Written in Python, based on Django.
  • 317133 lines of code
  • Fun libraries used:
    • Flot (for the résumé skillset charts)
  • Neat stuff I added:
    • A new, hideous design!
    • A hierarchical tagging system
    • A custom image resizing library. I couldn't find a use for the other ones out there.
    • The Konami Code. Try it, it's fun :-)
  • Stuff that's coming:
    • Search
    • Mobile image upload (snapshots)
    • The image gallery will be up as soon as the shots are done uploading.

Anyway, if you feel so inclined, please poke around and look for problems. I'll fix them as soon as I can.

An Update is Coming

August 10, 2010 12:16 +0000  |  Blogger Django PHP Python 1

For those who have been demanding that I post something, anything, (*cough* Noreen *cough*) I apologise for the delay, but it won't be long now. I've been using all this time to write a new version of my site, done up in Python/Django. The next version will be a watered-down version of this one (on account of the complete rewrite) but will grow with time.

I may also decide to abandon all attempts at making it pretty... 'cause well... I suck at that :-)

Facebook Page Syncronisation

January 03, 2010 12:07 +0000  |  Django Facebook Python Software TheChange.com Web Development 2

This is going to be a rather technical post, coupled with a smattering of rants about Facebook so those of you uninterested in such things might just wanna skip this one.

As part of my work on my new company, I'm building a syncroniser for status updates between Twitter, Facebook, and our site. Eventually, it'll probably include additional services like Flickr, but for now, I'm just focusing on these two external systems.

A Special Case

Reading this far, you might think that this isn't really all that difficult for either Twitter or Facebook. After all, both have rather well-documented and heavily used APIs for pushing and pulling data to and from a user's stream, so why bother writing about it? Well for those with my special requirements, I found that Facebook has constructed a tiny, private hell, one in which I was trapped for four days over the Christmas break. In an effort to save others from this pain, I'm posting my experiences here. If you have questions regarding this setup, or feel that I've missed something, feel free to comment here and I'll see what I can do for you.

So, lets start with my special requirements. The first stumbler was the fact that my project is using Python, something not officially supported by Facebook. Instead, they've left the job to the community which has produced two separate libraries with different interfaces and feature sets.

Second, I wasn't trying to syncronise the user streams. Instead, I needed push/pull rights for the stream on a Facebook Page, like those created for companies, politicians, famous people, or products. Facebook claims full support for this, but in reality it's quite obvious that these features have been crowbared into the overall design, leaving gaping holes in the integration path.

What Not to Do

  • Don't expect Facebook to do the right/smart thing. Everything in Facebookland can be done in one of 3 or 4 ways and none of them do exactly what you want. You must accept this.
  • Don't try to hack Facebook into submission. It doesn't work. Facebook isn't doing that thing that makes sense because they forgot or didn't care to do it in the first place. Accept it and deal. If you try to compose elaborate tricks to force Facebook's hand, you'll only burn 8 hours, forget to eat or sleep in the process and it still won't work.

What to Do

Step 1: Your basic Facebook App

If you don't know how to create and setup a basic canvas page in Django, this post is not for you. Go read up on that and come back when you're ready.

You need a simple app so for starters get yourself a standard "Hello World" canvas page that requires a login. You can probably do this in minifb, but PyFacebook makes this easy since it comes with handy Django method decorators:

# views.py
from django.http import HttpResponse, HttpResponseRedirect
import facebook

@facebook.djangofb.require_login()
def fbCanvas(request):
    return HttpResponse("Hello World")
Step 2: Ask the User to Grant Permissions

This will force the user to add your application before proceeding, which is all fine and good but that doesn't give you access to much of anything you want, so we'll change the view to use a template that asks the user to click on a link to continue:

# views.py
from django.shortcuts import render_to_response
from django.template import RequestContext
import facebook

@facebook.djangofb.require_login()
def fbCanvas(request):
    return render_to_response(
        "social/canvas.fbml",
        {},
        context_instance=RequestContext(request)
    )

Note what I mentioned above, that we're asking the user to click on a link rather than issuing a redirect. I fought with Facebook for a good few hours to get this to happen all without user-input and it worked... sometimes. My advice is to just go with the user-clickable link. That way seems fool-proof (so far).

Here's our template:

<!-- canvas.fbml -->
<fb:header>
    <p>To enable the syncronisation, you'll need to grant us permission to read/write to your Facebook stream.  To do that, just <a href="http://www.facebook.com/connect/prompt_permissions.php?api_key=de33669a10a4219daecf0436ce829a2e&v=1.0&next=http://apps.facebook.com/myappname/granted/%3fxxRESULTTOKENxx&display=popup&ext_perm=read_stream,publish_stream,offline_access&enable_profile_selector=1">click here</a>.
</fb:header>

See that big URL? It's option #5 (of 6) for granting extended permissions to a Facebook App for a user. It's the easiest to use and hasn't broken for me yet (Numbers 1, 2, 3 and 4 all regularly complained about silly things like not having the app instaled when this was not the case, but your milage may vary). Basically, the user will be directed to a page asking her to grant read_stream, publish_stream, and offline_access to your app on whichever pages or users she selects from the list of pages she administers. Details for modifying this URL can be found in the Facebook Developer Wiki.

Step 3: Understanding Facebook's Hackery

So you see how in the previous section, adding enable_profile_selector=1 to the URL will tell Facebook to ask the user to specify which pages to which she'd like to grant these shiny new permissions? Well that's nifty and all, but they don't tell you which pages the user selected.

When the permission questions are finished, Facebook does a POST to the URL specified in next=. The post will include a bunch of cool stuff, including the all important infinite session key and the user id doing all of this, but it doesn't tell you anything about the choices made. You don't even know what page ids were in the list, let alone which ones were selected to have what permissions. Nice job there Facebook.

Step 4: The Workaround

My workaround for this isn't pretty, and worse, depends on a reasonably intelligent end-user (not always a healthy assumption), but after four days cursing Facebook for their API crowbarring, I could come up with nothing better. Basically, when the user returns to us from the permissioning steps, we capture that infinite session id, do a lookup for a complete list of pages our user maintains and then bounce them out of Facebook back to our site to complete the process by asking them to tell us what they just told Facebook. I'll start with the page defined in next=:

# views.py
@facebook.djangofb.require_login()
def fbGranted(request):

    from cPickle import dumps as pickle
    from urllib  import quote as encode

    from myproject.myapp.models import FbGetPageLookup

    return render_to_response(
        "social/granted.fbml",
        {
            "redirect": "http://mysite.com/social/facebook/link/?session=%s&pages=%s" % (
                request.POST.get("fb_sig_session_key"),
                encode(pickle(FbGetPageLookup(request.facebook, request.POST["fb_sig_user"])))
            )
        },
        context_instance=RequestContext(request)
    )

# models.py
def FbGetPageLookup(fb, uid):
    return fb.fql.query("""
        SELECT
            page_id,
            name
        FROM
            page
        WHERE
            page_id IN (
                SELECT
                    page_id
                FROM
                    page_admin
                WHERE
                    uid = %s
            )
    """ % uid)

The above code will fetch a list of page ids from Facebok using FQL, and coupling it with the shiny new infinite session key, bounce the user out of Facebook and back to your site where you'll use that info to re-ask the user about which page(s) you want them to link to Facebook.

Step 5: Capture That page_id

How you capture and store the page id is up to you. For me, I had to create a list of organisations we're storing locally and let the user compare that list of organisations to the list of Facebook Pages and make the links appropriately. Your process will probably be different. Regardless of how you do it, just make sure that for every page you wish to syncronise with Facebook, you have a session_key and page_id.

Step 6: Push & Pull

Because connectivity with Facebook (and Twitter) is notonoriously flakey, I don't recommend doing your syncronisation in real-time unless your use-case demands it. Instead, run the code via cron, or better yet as a daemon operating on a queue depending on the amount of data you're playing with. However you do it, the calls are the same:

import facebook

# Setup your connection
fb = facebook.Facebook(settings.FACEBOOK_API_KEY, settings.FACEBOOK_SECRET_KEY)
infinitesessionkey = "your infinite session key from facebook"
pageid             = "the page id the user picked"

# To push to Facebook:
fb(
    method="stream_publish",
    args={
        "session_key": infinitesessionkey,
        "message":     message,
        "target_id":   "NULL",
        "uid":         pageid
    }
)

# To pull from Facebook:
fb(
    method="stream_get",
    args={
        "session_key": infinitesessionkey,
        "source_ids": pageid
    }
)["posts"]

Conclusion

And that's it. It looks pretty complicated, and... well it is. For the most part, Facebook's documentation is pretty thorough, it's just that certain features like this page_id thing appear to have fallen off their radar. I'm sure that they'll change it in a few months though, which will make my brain hurt again :-(

An Output Highlighter

November 13, 2009 17:51 +0000  |  Programming Python Software 0

I wrote something like this some time ago, but this version is much better, if only because it's in python. Basically, it's a script that highlights standard input based on arguments passed to it.

But how is that useful? Well imagine that you've dumped the contents of a file to standard output, maybe even piped it through grep, and/or sed etc. Oftentimes you're still left with a lot of text and it's hard to find what you're looking for. If only there was a way to highlight arbitrary portions of the text with some colour...

Here's what you do:

$ cat somefile | highlight.py some strings

You'll be presented with the same body of text, but with the word "some" highlighted everywhere in light blue and "strings" highlighted in light green. The script can support up to nine arguments which will show up in different colours. I hope someone finds it useful.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys,re

colours = [
    "\033[1;34m", # light blue
    "\033[1;32m", # light green
    "\033[1;36m", # light cyan
    "\033[1;31m", # light red
    "\033[1;33m", # yellow
    "\033[0;32m", # green
    "\033[0;36m", # cyan
    "\033[0;33m", # brown
    "\033[1;35m", # pink
    "\033[0m"     # none
]

args = sys.argv[1:]

# Strip out arguments exceeding the maximum
if len(args) > 9:
    print("\n%sWARNING: This script only allows for a maximum of 9 arguments.%s\n\n" % (colours[4], colours[9]), file=sys.stderr)
    args = args[0:8]

while True:
    line = sys.stdin.readline()
    colour = 0
    for arg in args:
        line = re.sub(
            r"(%s)" % (arg),
            "%s%s%s" % (colours[colour], "\g<1>", colours[9]),
            line
        )
        colour = colour + 1
    if line == '':
        break
    try:
        print(line.rstrip("\n"))
    except:
        pass

php.py

July 08, 2009 22:25 +0000  |  PHP Programming Python 0

I wrote something rather fun today and I thought that I'd share it here. It's a Python module that you can use to interact with PHP products. Specifically, it's a reproduction of PHP's http_build_query() and parse_ini_file() functions that act as PHP does according to PHP's own way of doing things.

This means that if you've written an API server (as we have) in PHP that makes use of things like the above, you can interact with it using Python as your scripting language with little effort.

Examples:

from php import parse_ini_file

config = parse_ini_file("/path/to/config.ini")
print config["sectionName"]["keyName"]

This would give you the value for keyName in the section called sectionName in your config.ini file.

from php import http_build_query

somedata = {
  "keyname": "valuename",
  "otherkey": 123,
  "anotherkey": [1,2,3,{"seven": "eight"}]
}
print http_build_query(somedata)

This would give you:

otherkey=123&keyname=valuename&anotherkey[1]=2&anotherkey[0]=1&anotherkey[3][seven]=eight&anotherkey[2]=3&

The code was fun to write, and I'm guessing that it'll be useful to others so I'm posting it here. If you do end up using it, lemme know by posting a comment here eh?

You can download it here: php.py.

When I mentioned this to some other coworkers, they pointed out that I'm not the only one trying to get some of PHP's odd functionality into Python. Another developer has mimicked PHP's serialize() functions in the form of a Python module. I wonder if there are any other cases where this kind of stuff might be useful.

Battery Level in Your Prompt with Python

May 17, 2009 07:31 +0000  |  Linux Python 1

In the midst of one of those "because I can" moods today, I wrote a fun Python script to get my battery status and colour-code it so it could be loaded into my prompt. I'm posting it here 'cause I think it's nifty:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import re

battery = "/proc/acpi/battery/BAT0"

def getMax(path):
    return getValueFromFile(path + "/info", "last full capacity")


def getRemaining(path):
    return getValueFromFile(path + "/state", "remaining capacity")


def getValueFromFile(name, value):
    f = open(name, "r")
    for line in f:
        remaining = re.match(r"^%s:\s+(\d+)" % (value), line)
        if remaining:
            return remaining.group(1)


def isCharging(path):
    f = open(path + "/state", "r")
    for line in f:
        key = re.match(r"^charging state:\s+charging", line)
        if key:
            return True


def render(path):

    level = int((float(getRemaining(path)) / float(getMax(path))) * 100)

    colour = ""
    if isCharging(path):
        colour = "\033[1;36m" # Cyan
    elif level < 25:
        colour = "\033[1;31m" # Red
    elif level < 50:
        colour = "\033[1;33m" # Yellow
    else:
        colour = "\033[1;32m" # Green

    print colour + str(level) + "%\033[0m",

render(battery)

Playing Catch Up With Opera, Choir, Secrets and Travel

March 05, 2009 07:11 +0000  |  Family Friends Japan Korea Linux Python Scrubby Travel 4

It's true. I'm still alive, though I couldn't blame you if you'd considered otherwise. I've been neglecting this blog of late. Actually, I've been neglecting most of my life lately but soon, very soon, I shall have a break and I wanted to get this Long List of Stuff out of the way before that happens so here goes:

Carmen

A little over a month ago, I attempted to expand my cultural horizons by taking in My First Opera at the Queen Elizabeth Theatre. I accompanied Margaret, Dianna, and Aisha to the show and like good opera-goers we dressed up pretty for the night, then quietly mocked the yahoos who felt that jeans and a tshirt was appropriate.

For my part, I can't say that I really enjoyed the opera. (Sorry Diana). I didn't hate it either though. Frankly, it didn't do much for me at all. I found much of the music frustratingly simple when compared to a symphony or even broadway show, and the characters completely unbelievable. The emotion they conveyed (quite brilliantly I admit) didn't make any sense when the story seemed so trivial. I guess Opera just isn't for me.

I still have trouble getting over the fact that they would hold something like an opera in a venue that doesn't really lend itself to acoustic projection. The QE Theatre, while quite functional as a normal theatre, doesn't hold a candle to the acoustics you find in The Orpheum, yet they hold rock concerts in the latter and opera in the former. This makes no sense to me.

Choir

Not too long after my night at the opera, I went to my first choir practise in years. Simple Gifts, a local amateur choir run by Ieva Wool and for the most part, I liked them. The people I sang with had talent, the director was patient and helpful and overall everyone in the room seemed to really enjoy the whole experience. The only negatives were the average age of the singers (~50ish) and the fact that the practise was held on Tuesday nights... I had no idea how tiring a regular weekday practise from 7:30 - 9:30 would be, but it was.

I had the opportunity to try out the choir for two practises before I decided whether or not I was "in" or not, and the decision of whether or not to keep going came down to a simple gut feeling: I was just too tired. That is, the idea of going to choir on Tuesday felt more like a responsibility ("you're going to like, this so you have to go") as opposed to a joy ("yay! choir!"). I chalked it up to the general energy level of the choir (dear gods I miss Mr. Rhan sometimes) and my own energy reserves at the end of my work day. I just couldn't give anymore, so I declined to join.

If my situations changes for the next "term", I'll drop in again and give it another go, but for now, I just didn't feel like I was getting what I needed out of it.

The Super Secret Project

My father is an Idea man. Much like myself, he has new ideas all the time, though the difference between us is that his ideas are usually profit-driven while mine remain the betterment of mankind-types. His latest idea however has been snowballing into a full-blown project and will likely launch this year. Through the life cycle of his this beast, he's been coming back to me asking questions about how he could do "x" and I would work out with him roughly how everything would work... well it's time, now he wants me to build it.

I've done some research and it looks like I'll be installing Gentoo Linux on one of these running a really cool Python script I wrote that captures mouse clicks and logs stuff to the database and then pushes said data over the Internet to a master server via one of these things. It's gonna be fun.

Korea and Japan

And now for the big one: I'm going to Korea on Saturday and then to Japan on the 14th, then home by the 22nd. It's gonna be frickin' cool. My friend Susan, who's currently teaching English in Daegu, Korea was looking for company for a Japan trip and I jumped at the chance (finances be damned!). The way I see it, Japan is too foreign a country for me to be comfortable exploring on my own, and frankly, few of my friends have the money or the interest in making the trip. This opportunity was too rare to pass up... and so I go!

It looks like th total cost of flights, trains and accommodation will be in the neighbourhood of $3000CAD which may sound crazy high but you have to remember that it is the other side of the world -- the two trans-Pacific flights alone make up 50% of that sum.

It'll be fun to hang with Susan though -- we never spent enough time together when we were both in Toronto, so this will give us time to catch up :-) She has her heart set on a traditional costuming thing that they do regularly in a park in Tokyo, and I'm really stoked about both riding the subway in there and visiting the Nintendo headquarters in Kyoto... no, I don't know if they have tours, but I don't care. I just want my picture in front of the Nintendo sign :-)

I'm currently taking orders for stuff people want me to bring back, so if you want on the list, just drop me a comment. Also, if you think that there's something I should see out that way, let me know and I'll try to add it to our itinerary. The cities I'll be in are: Seoul, Daegu (maybe), Tokyo, Kyoto, Okinawa City, and Naha.

Alright, I figure that makes up for my rather long absence. I'll try to be more studious when I'm blogging on the other side of the planet :-)