Blog

December 31, 2016 12:38 +0000  |  Family Grandpa Programming 1

I built a thing for my family this Christmas and I wanted to post about it briefly.

If you're one of the few people dedicated enough to follow this blog, you'll know that my grandfather died last year, and that he was sort of the family videographer. What you likely don't know however is that this year, on my trip home I acquired his entire collection of DVDs that he'd been accumulating over the years.

This some really old stuff:

  • Around the Christmas tree when I was 3 or 4 years old
  • My dog learning tricks for the first time
  • My parent's wedding
  • My graduation
  • My mother as a child in Romania
  • My grandparents, so much younger, with friends in Romania
  • My niece, Violet

It was an amazing collection spanning 4 generations over 39 DVDs, and I spent a few days on that trip home ripping every last one of the disks onto a portable hard drive so I could take the raw data home for a special project.

Well that project is now finished, so for those of you who don't care about the technical aspects, here's the link. I shared the URL with my family by email on Christmas day since I was on the other side of the world for the holiday festivities this year, but all in all, it seems to have gone over well.

My father has suggested that I expand on the collection with my own videos in the future -- I may just do that, though I'm more of a still photos guy. We'll see.

The Technical

This whole thing was a HUGE pain in the ass, so I want to document the process, perhaps if only for future websurfers looking to do something similar.

The Problem

The videos were in DVD format. Thankfully, it was digital, but it's certainly not web-friendly. The video data needed to be ripped from the disks and compressed into a web-friendly format that was high-quality enough to preserve the video, but in a file small enough to stream to Canada-quality internet connections.

Also, the DVDs were terribly organised and not indexed in any way. The disks often had multiple title tracks, sometimes duplicate tracks, and there were tracks that just contained garbage data.

Oh, and there was a time constraint. I only had the disks for a few days when I was in Canada. I wasn't going to take them back to the UK with me.

The Process

It was basically done in three stages:

Raw DVD > .iso file > .webm file

The .iso file step was just a clean & easy way to back up all of the DVDs without having to worry about accidentally missing something while I was hurriedly trying to get through them all in Canada. By turning 39 DVDs into 39 files on a USB drive, I could be sure that I wouldn't accidentally lose data during the ripping process.

As it turns out, this was a good plan, since it took a few weeks of tinkering with this project before I realised that some disks had multiple titles on them.

The creation of the .iso files was easy. I just put the disk in the USB DVD drive I brought with me and typed this:

$ dd if=/dev/dvd of=/path/to/usb/hard-drive/disk-00.iso

Waited about 20min, then took the disk out, and repeated this... 39 times.

The creation of the actual video file on the other hand was the big problem. There are lots of sites out there that claim to tell you how to do this, and very few of them have anything helpful. I think that this is because the end goal is rarely understood up front. Sometimes people are trying to encode DVDs into a high quality file for local playback, and the settings for that are rather different from what someone would want to do to encode for a web-friendly format.

There's also a wide variety of tools out there, most of which are buggy, unsupported, don't have a port for Gentoo, or just plain suck. The most common recommendation I found was for Handbrake, which is an impressive GUI for ripping videos but for me:

  • It didn't encode files that were high enough quality given the file size
  • It didn't make web-friendly formats. Even when you tick the box to make it web-friendly, the output file doesn't stream in Firefox. I didn't test other browsers.
  • It was terribly slow to find all the tracks, apply the settings I wanted and then wait to see if things panned out. There's no command-line interface to make things easier.

All of this lead to a lot of frustration and weeks of tinkering, finally leading me to a site that gave me the magic ffmpeg incantation to generate a web-friendly file:

$ ffmpeg \
  -i /path/to/input.mp4 \
  -vpre libvpx-720p \
  -pass 1 -passlogfile ffmpeg-18 -an -f webm \
  -y /path/to/output.webm && \
  ffmpeg -i \
  /path/to/input.mp4 \
  -vpre libvpx-720p \
  -pass 2 -passlogfile ffmpeg-18 -acodec libvorbis -ab 100k -f webm \
  -y /path/to/output.webm

Of course this assumed a .mp4 input file, and I wanted to rip straight from the .iso, so after much digging, I discovered that ffmpeg has a means of concatenating (chaining) video inputs and it can read straight from a DVD's .VOB file. With this nugget of knowledge, all I had to do was mount the .iso locally and compile a list of files conforming to this naming convention:

VIDEO_TS/VTS_01_#.VOB

With that information, I wrote a quick shell script that ended up generating a great big queue file of commands that look a lot like this:

ffmpeg -i \
'concat:/mnt/grandpa/18/VIDEO_TS/VTS_01_1.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_2.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_3.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_4.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_5.VOB' \
-vpre libvpx-720p -pass 1 -passlogfile ffmpeg-18 -an -f webm \
-y /home/daniel/Projects/Grandpa/htdocs/vid/18.webm && \
ffmpeg -i \
'concat:/mnt/grandpa/18/VIDEO_TS/VTS_01_1.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_2.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_3.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_4.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_5.VOB' \
-vpre libvpx-720p \
-pass 2 -passlogfile ffmpeg-18 -acodec libvorbis -ab 100k -f webm \
-y /home/daniel/Projects/Grandpa/htdocs/vid/18.webm

Unfortunately, ffmpeg doesn't really do threading very well, and the prevailing advice out there appears to be that you should just thread the process yourself rather than ask ffmpeg to try to use all your CPUs itself. For this bit, I wrote a very simple paralleliser in Python and magically, all of the cores on my super machine could crunch Grandpa's videos, 16 at a time.

Finally, I wrapped the whole thing in a simple script that mounted all of the .isos simultaneously and then ran the paralleliser, and ran that in a tmux session so I could get on a plane and Fly to Greece while my computer did its thing for two days.

While I was in Athens, I spent a day or two fiddling with the site itself, getting video.js to work the way I wanted it to and playing with Select2 to try and get an interface that the non-technical people in my family could follow. I wish I had better skills in this area 'cause frankly, the site is kinda ugly, but at least it's functional now.

So that's it. I hope that one day, someone will find this stuff useful. The ffmpeg incantations were especially difficult to find and assemble, so I figure that'll help someone eventually.

March 06, 2016 15:17 +0000  |  Christina Free Software Grandpa London Ripe NCC Software Violet 0

I've been putting off writing this because frankly, I look back on 2015 and it really doesn't stand out too much. Especially when you compare it to previous years. In 2014 for example, I visited Australia and New Zealand. In 2013 I moved in with Christina, and in 2012 my parents came to visit me and we traveled to Paris, London, and Dublin. 2015 had its ups and downs, but honestly, I feel like a lot less happened this past year.

Two big things happened though, but you'll have to keep reading if you can't remember what they were.

Travel

As best memory serves, I only did 5 bits of significant travel this year, and only one trip was New and Exciting. I guess I can't complain though, these were all good trips.

Brussels & FOSDEM

As I do every year now, I went to FOSDEM for the annual Free conference. If you've never been, I can't recommend it enough, not the least because you get to meet the people who build the tools you use every day and thank them in person. You also get to buy all kinds of cool stuff that helps support these projects, which doesn't suck either.

Madrid & Seville

The most exciting trip of the year for me, not the least because Inga and Gerardo were getting married in Seville! I took a couple weeks off and did a solo trip to Madrid for a week before heading down to Seville where I met Christina for the actual wedding. The event was so pretty and both cities were lovely. It was a fabulous trip.

Cardiff

Another important trip was to Cardiff, not because it was Cardiff (because frankly, I didn't much care for the town), but because it was for DjangoCon Europe and I was giving my first public workshop wherein I taught a small group of people for an hour or so complete with exercises and a presentation I put together in advance.

It turns out, I really like this sort of thing. Maybe I'll be able to do it as a proper job one day.

Vienna

Max and Julia, friends of Christina's from her law school days were getting married in a little town about two hours south of Vienna, so we flew out to the land of lederhosen and beer (seriously, there was a festival in town and everyone was out in their traditional garb) for a beautiful wedding in a castle.

The ceremony was long, and Catholic, and German, but the bride was gorgeous and the reception was excellent. Everyone had a lovely time.

Vancouver

The last big trip of the year was my annual visit back home to see friends & family. Violet is getting so big, and I worry that she really has no idea who I am. Like any 3 year old girl though, she's friendly and inclusive when it comes to strangers, so we still got along fine. I hope that one day we'll be able to stay in closer contact, but that's still a ways off.

It was also my last opportunity to see my grandfather alive.

Personal

Grandpa

My Grandpa died in 2015. It was harder on me than I'd expected, especially since he had spent much of the last few years essentially waiting to die. He'd become more isolated and frustrated as his body continued to fail him, and rarely if ever left the house. Still, I'm glad I had one more opportunity to spend time with him before he died, even if an unfortunately large portion of that time was spent in disjointed arguments and yelling about cable TV signals.

I wrote a eulogy for him, and then I rewrote it because the first one wasn't right. Every once in a while I re-read them.

Proposal

Before my grandfather died though, Christina and I were able to give him the news that she and I are getting married. I proposed on a warm August day, but Christina didn't get her ring until months later because I left that part up to her ;-) We ended up getting a modest white gold ring with a manufactured diamond for ethical reasons and Christina seems very happy with it.

Unfortunately, actually setting a date has been tough thus far, but only because our professional positions are a little chaotic at the moment. More on that later.

Moving to London

The biggest news of 2015, and probably the primary reason this year didn't involve all that much travel is that Christina and I packed up our lives in Amsterdam and moved to London. The idea was that she would start at an exciting new job, and I would find work somewhere in a Big City -- something I've missed a lot since moving to Amsterdam 5 years ago.

The visa application process was completely insane:

  • £956 application fee
  • £850 NHS surcharge
  • €500 expediting fee
  • £860 visa agent fee

On top of this we had to compile a ridiculous amount of information including a complete list of every out-of-country trip I've made in the last 5 years. That one took a lot of effort, but the number came down to 53 trips -- yes, I'm rather proud of that one.

We also had to find a flat and sign a one-year lease without actually seeing the apartment. Our new home is cold, damp, small, and on the ground floor. It used to be a stable. We're stuck here 'til November.

Professional

Tweetpile

It seems that every year that goes by, I adopt a new hobby project. In In 2013 I restarted Spirithunter (turns out Nintendo is going build a game just like it), in 2014 I had my father's RxLenses site, and in 2015 I started TweetPile, a project to help me collect all the tweets about anything in particular.

It took a lot of work, and I learnt a lot about asynchronous Python, but in the end, there didn't seem to be a lot of interest in it out there. The biggest problem is that people typically don't know what they want to collect on twitter until after the tweets have gone out. Twitter doesn't have a comprehensive historical API though, so you can't get all tweets in the past about "x", only stuff Twitter thinks is relevant. Long story short, TweetPile was fun to write, still works, but sadly isn't very popular.

RIPE Atlas Tools Magellan

After years of talking about it amongst the team and with the community, we embarked on actually writing a comprehensive command line toolkit for RIPE Atlas, the project I'd been labouring on ever since I started working there back in 2012.

This project was my baby, and I took point on it, dictating structure, choosing the license (GPL3!), and writing the lion's share of the code. I also worked with Andreas to adopt a proper pull-request-and-merge methodology, which was a nice change from how we had done things until then.

The result was a solid, thriving, community project that opens RIPE Atlas up to thousands of engineers all over the world. I'm really proud of this one.

Leaving RIPE

Of course, just as I started on an exciting new project, it was time for me to leave. I'd put in more than three years at the RIPE NCC, longer than any job I'd ever held, largely because I loved working for them.

It turns out that London has a lot of opportunities for Python nerds like me, so there were no shortage of jobs available. I had been spoilt by the NCC though, and wanted to continue working for a non-profit, so this tainted my search process. In the end, I ended up choosing to work for the British government. If you want to know how that's going though, you'll have to log in and see my private post about it ;-)

Conclusion

When I wrote this post for 2014, I talked about how I had big travel plans and possibly even dance classes in mind for 2015. In retrospect, I have no idea what I was thinking. I don't even remember what I was talking about then.

2016 promises to be interesting (new city, new friends, new job), but much of it is still up in the air. Christina is applying for a job in Edinburgh, and if she gets it, we're moving again -- as early as August. If she doesn't, then I have to figure out what I'm going to do about my work situation (more info in the private posts). This potential job also means that we're not likely to get married this year as there's no sense in planning a wedding in Greece when you're not even sure where you'll be living in a couple months.

I've also joined a choir here in London which is pretty awesome, and while I've managed to lose a little weight in my last year in Amsterdam, it's creeping back up living here in the land of fried and salted everything.

I have no idea which way things are going to go, and to be honest, I'm annoyed that I don't find this exciting.

September 20, 2015 23:01 +0000  |  Family Grandpa 0

I want to take a minute to rewrite my eulogy for my grandfather. There was something bugging me about that first draft, the one that was eventually read at his funeral that didn't sit well with me, and a few days later Jane finally helped me figure it out: I wasn't writing it for me.

Instead, I wrote it in an attempt to reflect how we all interacted with him, and for such a polarising person, I simply don't have the talent to express something that even-handed as well as talk about what he meant to me. So that eulogy feels empty to me. This one is better.

I loved my grandpa. He was a difficult man to love sometimes, but I loved him anyway. He was, to me at least, the Caretaker of the family. The one who looked after me and helped me on my path -- whatever it was -- but he always required convincing.

Taking on a new career? I had to prove to him why this path was good for me. Moving to a new city? A new country? He wanted to know what kind of work I would find there, and when I was going to meet a nice girl and get married.

"Women want to see three keys" he used to say. "One for the house, one for the car, and one for the safety deposit box". My grandfather was very old-fashioned, and sexist, often dismissive, and almost always self-important and a little bit delusional about how the world worked, but he loved me, I'm 100% sure of it. How? because the man slipped me 100 bucks whenever he could.

This is how Grandpa showed you how he felt: he helped you in whatever way he could. He didn't have a lot of money, but he knew that when I was getting started in life, I had a lot less than he did. He would give me a hug whenever I'd come to visit, then offer to shake my hand -- a brown bill squeezed between his fingers. It didn't happen every time, just once in a while, when he could afford it, and no amount of objections would be accepted. He wanted to help his grandson and that was the end of it.

He spent much of his life compiling video footage of the family. I've seen video clips of my mother as a child, a teenager, an adult, and a mother. There's a video of my brother showing off his basketball skills, of a big Easter dinner celebration, of my brother and me opening presents on Christmas Day. My grandfather would watch these videos on his own time, whenever the mood struck him, first on high-8, then on VHS, finally on DVD, he migrated all of it by hand. He would insist on sharing them with girlfriends I brought over. It was his way of preserving the family, of remembering the life he'd led.

I'm going to miss my grandpa. He was crochety and pointlessly argumentative, and in his old age, even abusive, but even with all of that, I'm going to miss him because he was a good person who loved me and only wanted to help.

September 10, 2015 22:38 +0000  |  Family Grandpa

Grandpa was an impossible man -- both in that he was difficult to be around at times and in the amazing life he led.

Here was a man who was not only 100% confident that he had the answers to everything, but he was going to do you the immense favour of pointing out everything you're doing wrong -- you know, for your benefit.

I remember a particular Christmas gathering at which he persisted in his argument with his two atheist grandchildren that "God" created everything: "Who made this?" he would ask, "and who made this?", repeatedly pointing to random objects in the room. There was no winning an argument with him, you could only hug him and say "I love you Grandpa".

This would usually buy you a few minutes.

One of my earliest memories as a child is that of my father's disapproval of my grandfather's spending money on my brother and me. His refrain "Money is for spending!" will forever be a part of me. I think that deep down, Grandpa was a bit of a hedonist, but it was the simple things in life that did it for him. He loved his car, his boat, that obnoxious talking fish, and of course, he loved his family.

It's easy to forget in this era of smart phones, but we all owe a great debt to him for the hours and hours of home videos he took of all of us as we grew up. There are videos of my mother as a child, my parents getting married, the many barbecues and Easter gatherings -- all painstakingly preserved, transferred between formats over the years. This was a labour of love for him: the preservation of memory for three generations. How sad it is that he should leave us all such a gift when he himself appears so seldom in the frame.

Take a moment to consider what he accomplished in his lifetime:

  • He escaped Communism with his family to start a new life in a country where he didn't even speak the language.
  • He then proceeded to found multiple businesses across Canada employing dozens of people.
  • He supported every member of the family, either financially, with skills training, or simply with a place to sleep when one of us needed it.

He was undoubtedly an egoist and a pain in the ass, but he was also unabashedly generous and unconditionally loving.

Grandpa was an unyielding force in this world, and we are all so fortunate to have had the opportunity to be part of his life. He will forever be an inspiration to me.

True to form, Grandpa died on his schedule and no one else's. The world may be a lot quieter without him in it, but there's no doubt in my mind that it is also greatly diminished.

He'll wait for us right here.

My grandfather died peacefully in his sleep on Sunday night. He is survived by his wife, two daughters, three grandsons, and one great-granddaughter. He was 91.