Blog

Grandpa's Archive

December 31, 2016 12:38 +0000  |  Family Grandpa Programming 1

I built a thing for my family this Christmas and I wanted to post about it briefly.

If you're one of the few people dedicated enough to follow this blog, you'll know that my grandfather died last year, and that he was sort of the family videographer. What you likely don't know however is that this year, on my trip home I acquired his entire collection of DVDs that he'd been accumulating over the years.

This some really old stuff:

  • Around the Christmas tree when I was 3 or 4 years old
  • My dog learning tricks for the first time
  • My parent's wedding
  • My graduation
  • My mother as a child in Romania
  • My grandparents, so much younger, with friends in Romania
  • My niece, Violet

It was an amazing collection spanning 4 generations over 39 DVDs, and I spent a few days on that trip home ripping every last one of the disks onto a portable hard drive so I could take the raw data home for a special project.

Well that project is now finished, so for those of you who don't care about the technical aspects, here's the link. I shared the URL with my family by email on Christmas day since I was on the other side of the world for the holiday festivities this year, but all in all, it seems to have gone over well.

My father has suggested that I expand on the collection with my own videos in the future -- I may just do that, though I'm more of a still photos guy. We'll see.

The Technical

This whole thing was a HUGE pain in the ass, so I want to document the process, perhaps if only for future websurfers looking to do something similar.

The Problem

The videos were in DVD format. Thankfully, it was digital, but it's certainly not web-friendly. The video data needed to be ripped from the disks and compressed into a web-friendly format that was high-quality enough to preserve the video, but in a file small enough to stream to Canada-quality internet connections.

Also, the DVDs were terribly organised and not indexed in any way. The disks often had multiple title tracks, sometimes duplicate tracks, and there were tracks that just contained garbage data.

Oh, and there was a time constraint. I only had the disks for a few days when I was in Canada. I wasn't going to take them back to the UK with me.

The Process

It was basically done in three stages:

Raw DVD > .iso file > .webm file

The .iso file step was just a clean & easy way to back up all of the DVDs without having to worry about accidentally missing something while I was hurriedly trying to get through them all in Canada. By turning 39 DVDs into 39 files on a USB drive, I could be sure that I wouldn't accidentally lose data during the ripping process.

As it turns out, this was a good plan, since it took a few weeks of tinkering with this project before I realised that some disks had multiple titles on them.

The creation of the .iso files was easy. I just put the disk in the USB DVD drive I brought with me and typed this:

$ dd if=/dev/dvd of=/path/to/usb/hard-drive/disk-00.iso

Waited about 20min, then took the disk out, and repeated this... 39 times.

The creation of the actual video file on the other hand was the big problem. There are lots of sites out there that claim to tell you how to do this, and very few of them have anything helpful. I think that this is because the end goal is rarely understood up front. Sometimes people are trying to encode DVDs into a high quality file for local playback, and the settings for that are rather different from what someone would want to do to encode for a web-friendly format.

There's also a wide variety of tools out there, most of which are buggy, unsupported, don't have a port for Gentoo, or just plain suck. The most common recommendation I found was for Handbrake, which is an impressive GUI for ripping videos but for me:

  • It didn't encode files that were high enough quality given the file size
  • It didn't make web-friendly formats. Even when you tick the box to make it web-friendly, the output file doesn't stream in Firefox. I didn't test other browsers.
  • It was terribly slow to find all the tracks, apply the settings I wanted and then wait to see if things panned out. There's no command-line interface to make things easier.

All of this lead to a lot of frustration and weeks of tinkering, finally leading me to a site that gave me the magic ffmpeg incantation to generate a web-friendly file:

$ ffmpeg \
  -i /path/to/input.mp4 \
  -vpre libvpx-720p \
  -pass 1 -passlogfile ffmpeg-18 -an -f webm \
  -y /path/to/output.webm && \
  ffmpeg -i \
  /path/to/input.mp4 \
  -vpre libvpx-720p \
  -pass 2 -passlogfile ffmpeg-18 -acodec libvorbis -ab 100k -f webm \
  -y /path/to/output.webm

Of course this assumed a .mp4 input file, and I wanted to rip straight from the .iso, so after much digging, I discovered that ffmpeg has a means of concatenating (chaining) video inputs and it can read straight from a DVD's .VOB file. With this nugget of knowledge, all I had to do was mount the .iso locally and compile a list of files conforming to this naming convention:

VIDEO_TS/VTS_01_#.VOB

With that information, I wrote a quick shell script that ended up generating a great big queue file of commands that look a lot like this:

ffmpeg -i \
'concat:/mnt/grandpa/18/VIDEO_TS/VTS_01_1.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_2.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_3.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_4.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_5.VOB' \
-vpre libvpx-720p -pass 1 -passlogfile ffmpeg-18 -an -f webm \
-y /home/daniel/Projects/Grandpa/htdocs/vid/18.webm && \
ffmpeg -i \
'concat:/mnt/grandpa/18/VIDEO_TS/VTS_01_1.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_2.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_3.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_4.VOB|/mnt/grandpa/18/VIDEO_TS/VTS_01_5.VOB' \
-vpre libvpx-720p \
-pass 2 -passlogfile ffmpeg-18 -acodec libvorbis -ab 100k -f webm \
-y /home/daniel/Projects/Grandpa/htdocs/vid/18.webm

Unfortunately, ffmpeg doesn't really do threading very well, and the prevailing advice out there appears to be that you should just thread the process yourself rather than ask ffmpeg to try to use all your CPUs itself. For this bit, I wrote a very simple paralleliser in Python and magically, all of the cores on my super machine could crunch Grandpa's videos, 16 at a time.

Finally, I wrapped the whole thing in a simple script that mounted all of the .isos simultaneously and then ran the paralleliser, and ran that in a tmux session so I could get on a plane and Fly to Greece while my computer did its thing for two days.

While I was in Athens, I spent a day or two fiddling with the site itself, getting video.js to work the way I wanted it to and playing with Select2 to try and get an interface that the non-technical people in my family could follow. I wish I had better skills in this area 'cause frankly, the site is kinda ugly, but at least it's functional now.

So that's it. I hope that one day, someone will find this stuff useful. The ffmpeg incantations were especially difficult to find and assemble, so I figure that'll help someone eventually.

He Wanted to Help

September 20, 2015 23:01 +0000  |  Family Grandpa 0

I want to take a minute to rewrite my eulogy for my grandfather. There was something bugging me about that first draft, the one that was eventually read at his funeral that didn't sit well with me, and a few days later Jane finally helped me figure it out: I wasn't writing it for me.

Instead, I wrote it in an attempt to reflect how we all interacted with him, and for such a polarising person, I simply don't have the talent to express something that even-handed as well as talk about what he meant to me. So that eulogy feels empty to me. This one is better.

I loved my grandpa. He was a difficult man to love sometimes, but I loved him anyway. He was, to me at least, the Caretaker of the family. The one who looked after me and helped me on my path -- whatever it was -- but he always required convincing.

Taking on a new career? I had to prove to him why this path was good for me. Moving to a new city? A new country? He wanted to know what kind of work I would find there, and when I was going to meet a nice girl and get married.

"Women want to see three keys" he used to say. "One for the house, one for the car, and one for the safety deposit box". My grandfather was very old-fashioned, and sexist, often dismissive, and almost always self-important and a little bit delusional about how the world worked, but he loved me, I'm 100% sure of it. How? because the man slipped me 100 bucks whenever he could.

This is how Grandpa showed you how he felt: he helped you in whatever way he could. He didn't have a lot of money, but he knew that when I was getting started in life, I had a lot less than he did. He would give me a hug whenever I'd come to visit, then offer to shake my hand -- a brown bill squeezed between his fingers. It didn't happen every time, just once in a while, when he could afford it, and no amount of objections would be accepted. He wanted to help his grandson and that was the end of it.

He spent much of his life compiling video footage of the family. I've seen video clips of my mother as a child, a teenager, an adult, and a mother. There's a video of my brother showing off his basketball skills, of a big Easter dinner celebration, of my brother and me opening presents on Christmas Day. My grandfather would watch these videos on his own time, whenever the mood struck him, first on high-8, then on VHS, finally on DVD, he migrated all of it by hand. He would insist on sharing them with girlfriends I brought over. It was his way of preserving the family, of remembering the life he'd led.

I'm going to miss my grandpa. He was crochety and pointlessly argumentative, and in his old age, even abusive, but even with all of that, I'm going to miss him because he was a good person who loved me and only wanted to help.

Impossible

September 10, 2015 22:38 +0000  |  Family Grandpa

Grandpa was an impossible man -- both in that he was difficult to be around at times and in the amazing life he led.

Here was a man who was not only 100% confident that he had the answers to everything, but he was going to do you the immense favour of pointing out everything you're doing wrong -- you know, for your benefit.

I remember a particular Christmas gathering at which he persisted in his argument with his two atheist grandchildren that "God" created everything: "Who made this?" he would ask, "and who made this?", repeatedly pointing to random objects in the room. There was no winning an argument with him, you could only hug him and say "I love you Grandpa".

This would usually buy you a few minutes.

One of my earliest memories as a child is that of my father's disapproval of my grandfather's spending money on my brother and me. His refrain "Money is for spending!" will forever be a part of me. I think that deep down, Grandpa was a bit of a hedonist, but it was the simple things in life that did it for him. He loved his car, his boat, that obnoxious talking fish, and of course, he loved his family.

It's easy to forget in this era of smart phones, but we all owe a great debt to him for the hours and hours of home videos he took of all of us as we grew up. There are videos of my mother as a child, my parents getting married, the many barbecues and Easter gatherings -- all painstakingly preserved, transferred between formats over the years. This was a labour of love for him: the preservation of memory for three generations. How sad it is that he should leave us all such a gift when he himself appears so seldom in the frame.

Take a moment to consider what he accomplished in his lifetime:

  • He escaped Communism with his family to start a new life in a country where he didn't even speak the language.
  • He then proceeded to found multiple businesses across Canada employing dozens of people.
  • He supported every member of the family, either financially, with skills training, or simply with a place to sleep when one of us needed it.

He was undoubtedly an egoist and a pain in the ass, but he was also unabashedly generous and unconditionally loving.

Grandpa was an unyielding force in this world, and we are all so fortunate to have had the opportunity to be part of his life. He will forever be an inspiration to me.

True to form, Grandpa died on his schedule and no one else's. The world may be a lot quieter without him in it, but there's no doubt in my mind that it is also greatly diminished.

He'll wait for us right here.

My grandfather died peacefully in his sleep on Sunday night. He is survived by his wife, two daughters, three grandsons, and one great-granddaughter. He was 91.