Blog

November 15, 2010 22:19 +0000  |  Drupal Programming Software 14

I've been doing Drupal development on-and off for nearly three years now and it's always been frustrating. I'm a pretty vocal and animated kind of person too, so my co-workers soon came to know me as the anti-Drupal guy, which can be pretty rough when your employer has chosen to standardise on the platform. Now that I'm finally out of the Drupal world, I wanted to write a little about the platform, specifically speaking to its weaknesses and failures.

My hope here is two fold: (a) that this post serve as a means of communicating to the thousands of frustrated developers out there that they're not alone in their pain, and (b) that perhaps some of this post will help development shops choose Drupal where appropriate and other technologies when it is not.

For the Drupal fan(girl|boy)s, I ask only that you try to read this with an open and constructive mind. While I may rant and curse about Drupal in my Twitter feed, I've tried very hard to make this an unemotional, hopefully useful post about something I've spent a lot of time thinking about and working with.

Drupal Centricism

Drupal Ideology

It seems to be a mantra within the community: "You don't even need to write code". The Drupal ideology is user-centric, choosing ease-of-use over performance at every turn. There's nothing wrong with this of course, so long as your goal is to let unskilled people make websites. However if your priority is a performant application capable of handling a lot of traffic, you're going to have a number of problems.

Some examples of prioritising user-focus over performance:

  • Silent failures are the bane of any developer's existence. It's important to know when a variable isn't defined, or that writing a record to the database failed, or that a file didn't upload properly. Drupal suppresses such messages by default, and as a result nearly every contrib module in the community is so riddled with errors and warnings that development with these messages enabled is near impossible.
  • Views, the de-facto standard way to store and retrieve data from your database, writes queries to the database, so that in order to perform a query against the database, you must first fetch the query from the database. Similar inefficiencies can be found in other "standard" modules like CCK and Panels.
  • Drupal relies almost entirely on caching in order to function at all. Without caching, a method usually reserved for high to extreme traffic situations, Drupal can't handle even a small number of concurrent visitors. Indeed, some projects I've seen have taken more than 10minutes to load a single page, even in development where there was only one connection in use.

Drupal Magic

It's a term celebrated by many in the community. The idea being that Drupal does a mountain of work for you, so you don't have to worry about it. The only problem is that when you're trying to build a finely-tuned application, most of this magic either gets in the way, or even works against you. You get 80% of the way there with Drupal and its contrib modules, and then spend three months fighting the whole application, undoing the damage it's done, just to get what you need out of your website.

The hook-dependent system requires and fosters this anti-pattern. Re-using code often means unpredictable, site-wide changes. A property is written in module X, overwritten in module Y, and altogether removed in module Z, and there's no way to be certain that these functions will execute in a predictable order.

This problem is notably worse when it comes to new developers on a project, since they will undoubtedly not be privy to the magic that is running under the hood, and will have a difficult time discovering it on their own. To those who will answer this with "the project simply needs better documentation", I respectfully suggest that a good code base is easy to understand, and doesn't require a manual that is usually out of date.

To work with Drupal Magic is to attempt to produce useful code against an unordered, uncontrolled, grep-to-find-what-is-going-on-dependent architecture.

Drupal Community

For all the victories in community engagement Drupal has achieved (a massive, diverse and engaged membership), it's the glaring failures that make the whole project a miserable situation for developers. I've already mentioned the standardising on inefficient modules, but I haven't talked about the mountains of really horribly written code yet. Drupal Core, for what it does, is pretty efficient, but too many contrib modules are written by inexperienced developers, or are simply incapable of scaling to enterprise-level capacity. The result of this is that non-developers (managers, sometimes even clients) will point to the functionality of module X and insist: "don't redesign the wheel, just use that", and you spend the next three weeks trying to work around the poor design of said module, eventually being forced to write garbage that talks to garbage.

Often the perceived strength of the community is Drupal's greatest weakness. Drupal is promoted based on its theoretically infinite feature set, but the reality is that in order to use every one of those contrib modules in your site, the memory footprint will be massive, the stability suspect, and the performance abysmal. And gods help you if you try this on a site with millions of users or a similar number of content nodes.

Drupal Establishment

None of this is a problem however if Drupal is used where its features and shortcomings are both understood and accepted as the nature of the platform. Drupal is a great tool in some situations and a horrible burden in others. Sadly, this has not yet sunk in with many of the decision-makers in the web development community. Drupal is being used and promoted as a solution hammer, with every potential development project, a Drupal-shaped nail.

This has a number of negative outcomes, the most dangerous of which is a lack of skill diversity in developers. Companies that insist on Drupal-centric development are in fact promoting ignorance of alternatives that might do a better job and that hurts everyone. Unless developers at these companies take it upon themselves to spend time outside of their 8-12 hour work day to write code for a different platform or language, this Drupal dependency will force their non-Drupal skills to atrophy, limiting their ability to produce good code in the future.

Conclusion

I'm finally at the end of my admittedly unenthusiastic involvement in the Drupal community. Whether the Drupal shops out there read this isn't really up to me, but I hope that this manages to help some people re-evaluate their devotion to the platform. Comments are welcome, so long as they're constructive (I moderate everything), but I'm not going to get into a shouting match on the Internet. If you think I'm wrong, we can talk about it in 5 years.

June 15, 2009 19:07 +0000  |  Activism Drupal Free Software Linux PHP Software Technology Work [at] Play 0

I attended my first ever OpenWeb conference yesterday and as per company policy, I have to report on and share what I learnt, so what better way to do so then to make a blog post for all to read?

General

OpenWeb is awesome. It's a conference where people from all over the world come to talk about Open design and communication and hopefully, learn to build a better web in the process. Attendees include programmers, entrepreneurs, designers, activists and politicians all with shared goals and differing skillsets. I shook hands with Evan Prodromou, the founder of identi.ca and WikiTravel, heard talks from the guys who write Firefox and Thunderbird as well as the newly-elected representative for the Pirate Party in the European Parliament, Rickard Falkvinge. All kinds of awesome I tell you.

Rickard Falkvinge: Keynote - On the Pirate Party

Founder of the Pirate Party in Sweden and now a representative in the European Parliament (thanks to proportional representation), Falkvinge was a passionate and eloquent speaker who covered the history of copyright, the present fight for greater control of so-called intellectual property and more importantly the far-reaching and very misunderstood effects of some of the legislation being passed to "protect" copyright holders while eliminating privacy rights for the public.

The talk was very in depth and difficult to cover in a single post so I encourage you to ask me about it in person some time. For the impatient though, I'll try to summarise:

The copyright debate isn't about downloading music, that's just a byproduct of the evolution of technology. As the printing press gave the public greater access to information, so has the Internet managed to disperse that information further. The problem is now that the changing landscape has rendered certain business models ineffective, these business are fighting to change our laws to preserve said model rather than change with the times. Ranging from the frustratingly shortsighted attempts to ban technologies that further file sharing (legal or otherwise) to the instant wire tapping on every Internet connection (and by extension phone call) of every free citizen without a warrant, many of these changes are very, very scary.

"All of this has happened before, and it will happen again" he said. Every time a technological advancement creates serious change for citizen empowerment in society, the dominant forces in that society mobilise to crush it. The Catholic church, gatekeepers of the lion's share of human knowledge at the time actively worked to ban the printing press. They succeeded (if you can believe it) in France in 1535. This time, it's the media companies and they're willing to do anything, including associating file sharing with child pornography and terrorism to do it. Falkvinge's Pirate party is becoming the beachhead in the fight for copyright reform. Now the party with the largest youth delegation (30%!) in Sweden, they are working to get the crucial 4% of the seats in Parliament they need to hold the balance of power and they need your help. He'd like you to send the party 5€ or 10€ per month and I'm already on board.

Angie Byron: Keynote - Women in Open Source

Those of you who know me, know that I can get pretty hostile when it comes to treating women like a special class of people (be the light positive or negative) so I was somewhat skeptical about this one. Thankfully, I was happy to hear Byron cover a number of issues with the Free software community ranging from blatant sexism (CouchDB guys... seriously?) to basic barriers to entry for anyone new to a project. There were a lot of really helpful recommendations to people wanting to engage 100% of the community rather than just one half or the other.

Blake Mizerany: Sinatra

Sinatra is a Ruby framework that went in the opposite direction of things like my beloved Django or Ruby's Rails. Rather than hide the nuts and bolts of HTTP from the developer, Sinatra puts it right out there for you. Where traditional frameworks tend to muddle GET, POST, PUT, and DELETE into one input stream, this framework structures your whole program into blocks a lot like this:

  require 'rubygems'
  require 'sinatra'
  get '/hi' do
    "Hello World!"
  end

That little snipped up there handles the routing and display for a simple Hello World program. Sinatra's strength is that it's simple and elegant. It lets you get at the real power at the heart of HTTP which is really handy, but from what I could tell in the presentation, there's not a lot available outside of that. Database management is done separately, no ORM layer etc. etc. It's very good for what it does, but not at everything, which (at least in my book) makes it awesome.

Ben Galbraith and Dion Almaer: Mozilla Labs

These are the guys who make the Cool New Stuff that comes out of Mozilla. You know those guys, they write a nifty web browser called "Firefox", I'm sure you've heard of them.

Mozilla Labs is where the smart nerds get together to build and experiment with toys that will (hopefully) eventually make it into a finished product. Sometimes that product is an add-on or plug-in, other times it's an entirely new project. It's all about how useful something is to the public. And as always, the code is Free. You may have even heard of Ubiquity, an extension to Firefox that promises to reshape how we use a web browser... they're working on that.

This time through, they were demoing Bespin, a code editor in your web browser. Imagine opening a web browser, going to a page and doing your development there: no need for a local environment, but without the usual disadvantages of aggravating lag or difficult, text-only interface. Now imagine that you can share that development space with someone else in real time and that you can be doing this from your mobile device on a beach somewhere. Yeah, it's that awesome.

We watched as they demoed the crazy power that is the <canvas /> tag by creating a simple text editor, in Javascript right there in front of us... with about 15 lines of code. Really, really impressive.

David Ascher: Open Messaging on the Open Internet

Ascher's talk on Open Messaging was something I was really interested in since I've been actively searching for information on federated social networking for a while now. The presentation was divided into two parts: half covering the history of email and it's slow deprecation in favour of a number of different technologies as well as how people are using it in ways never intended for the architecture. Major problems with the protocol itself were touched on, as well as an explanation about how some of the alternatives out there are also flawed.

He then went on to talk about Mozilla Thunderbird 3 and the variety of cool stuff that's happening with it. "Your mail client knows a lot about you" he says "but until now, we haven't really done a lot with it". Some of the new features for Thunderbird 3 include conversation tracking (like you see in Gmail), helping you keep track of what kinds of email you spend the most time on, who you communicate with most etc. and even statistical charts about what time of day you use mail, what kind of mail you send and to whom how often. It's very neat stuff. Add to this the fact that they've completely rewritten the plug-in support, so new extensions to Thunderbird mean that your mail client will be as useful as you want it to be.

Evan Prodromou: Open Source Microblogging with Laconica

Up until this talk (and with the exception of Falkvinge's keynote), I'd been interested, but not excited about OpenWeb. Prodromou's coverage of Laconica changed all of that.

Founder of WikiTravel and one of the developers on WikiMedia (the software behind Wikipedia), Prodromou has built a federated microblogging platform called Laconica. Think Twitter, but with the ability for an individual to retain ownership of his/her posts and even handle distribution -- with little or no need for technical knowledge required. Here, I made you a diagram to explain:

Federated Laconica vs. Monolithic Twitter
Federated Laconica vs. Monolithic Twitter

Here's how it is: whereas Twitter is a single central source of information, controlled by a single entity (in this case, a corporation), Laconica distributes the load to any number of separate servers owned by different people that all know how to communicate. Where you might be on a server in Toronto, hosted by NetFirms, I could be using a Laconica service hosted by Dreamhost in Honolulu. My posts go to my server, yours go to yours, and when my Twitter client wants to fetch your posts, it talks to NetFirms and vice versa.

The advantages are clear:

  1. Infinite scalability: Twitter's monolithic model necessitates the need for crazy amounts of funding and they still don't have a profit model to account for those costs. Laconica on the other hand means that the load is distributed across potentially millions of hosts (much like the rest of the web).
  2. You control your identity, not a private corporation.

The future is where it gets really exciting though. By retaining ownership of your identity and data, you can start to attach a variety of other data types to the protocol. For the moment, Laconica only supports twitter-like messages, but they're already expanding into file-sharing as well. You'll be able to attach images, video and music files, upload them to your server and share them with whomever is following you. After that, I expect that they'll expand further to include Flickr-like photo streams, Facebook-like friendships and LiveJournal-like blog posts. These old, expensive monolithic systems are going away. In the future we'll have one identity, in one place, that we control that manages all of the data we want to share with others.

Really, really cool stuff.

I went home that night and signed up as a developer on Laconica. I've downloaded the source and will experiment with it this week before I take on anything on the "to do " list. I intend on focusing on expanding the feature set to include stuff that will deprecate the monolithic models mentioned above... should be fun :-)

Drupal Oops

I closed out the evening with some socialising in the hallway and some ranting about how-very-awesome Laconica was to my coworker Ronn, who showed up late in the day. He wandered off in search of my other colleagues and I followed after finishing a recap with Karen Quinn Fung a fellow transit fan and Free software fan. Unfortunately though, I wasn't really paying attention to where Ronn was going, I just followed out of curiosity. It turns that out I had stumbled into a Drupal social where I was almost immediately asked: "so, how do you use Drupal and how much do you love it?" by the social organiser. James gave me a horrified "what the hell are you doing here" look and searching for words, I said something to the effect of "Um, well, I was pretty much just dropping in here looking for my co-workers... oh here they are! -- I like Drupal because it makes it easy for people to make websites, but I don't really use it because it gets in my way. I prefer simple, elegant solutions and working around something just to get it to work is too aggravating." Considering the company, my response was pretty well received. I backed out quietly at the earliest opportunity :-)

So that was OpenWeb, well half of it anyway. I only got a pass for the Thursday. I can't recommend it enough though. Really interesting talks and really interesting people all over the place. I'll have to make sure that I go again next year.

June 27, 2008 07:52 +0000  |  CCTV Drupal Employment Family The Toronto Public Space Committee Work [at] Play 0

I'm trying to post more here lately as it would seem that since moving to Vancouver, my posts have been more and more sparsely scattered about the month. To that end, here's a quickie post regarding my relatively good day:

I had my review today. Good news: they like my work, and they're giving me a raise (w00t!) The only somewhat negative thing the boss mentioned was how I didn't know enough about Drupal yet. I can understand her position really, I mean, a large portion of our legacy code is in that horrible framework, so it only makes sense that as a senior developer, I know my way around it. I guess I'll just have to take a deep breath, bash my head in with a crowbar, and work from there. ^_^

Anyway, aside from the good review, I found a new sandwich shop in the area that serves giganimous sandwiches, and then discovered some left over birthday cake in the office fridge. This, coupled with the fact that I got a big chunk of work done today (and documented!) made me happy with my current form of employment.

I also fielded a 1hour call with a University student out of Windsor, Ontario to give him some background on the TPSC's CCTV campaign back when I was running the show. That was a bit of nostalgic fun -- kinda like talking to the press, but you can be a little more candid since you know that you're not talking to the uninformed public, but rather a well-read academic.

And then, to top it all off, I came home to a clean apartment AND new groceries in the fridge! Butthead had been hard at work and it showed. Having a roomate might not be so bad after all ;-) He's making a lot of progress with his own life lately though. I'm really quite proud of him.

March 12, 2008 21:25 +0000  |  Drupal Programming 1

I'm finally understanding what it's like to write code for Drupal. Essentially, you have to imagine the least efficient way of doing something, then look for how to do that... 'cause that's exactly what Drupal does.

I mean, why would anyone want to create additional columns to a user table when they could just put a bunch of serialised data into a string in the user table? It's just so easy! Why didn't I think of that?

Oh, right, 'cause I'm not insane.

January 31, 2008 05:14 +0000  |  Drupal Employment Perl Reinvent Work [at] Play 8

Those of you who aren't privy to my less-than-public posts, you might wonder what the heck happened to Reinvent and why I'm suddenly talking about The Donat Group. Obviously, it's probably a Bad Idea to talk about the nuts and bolts of it all publically, but here's the gist in my generation's favourite format: point form.

  • Reinvent is a fun company with lots of smart people
  • It's also run by management that feels that expanding their business into Software patents makes for an acceptable business model.
  • I disagreed. I've worked for a lot of companies who did things I felt were Not Good, but there are a few things I won't do... supporting software patents is one of them.
  • So I went looking for a company whose morals where more in tune with my own and I found one in The Donat Group.
  • I handed in my resignation, offered to work 2weeks before I left and they declined. I guess the brass weren't happy with me.
  • I started at Donat today and they seem pretty cool. Very dot.com startup-like but with actual paying clients.
  • They like Drupal though, which from what I've seen, is really not good. My first impressions of Drupal is that it's a bunch of non-programmers pretending that they can write real software. Function and variable names bleeding into everything else, no objects, and horror of all horrors: their coding standard requires a two-space indent.

I made a work around for the back-assward indenting thing though. I call it undrupal. Basically it uses a regular expression to strip out the space-based indents and replace them with tabs. Then you can write your code in a sane environment, then use the same script to re-crazy it with spaces again.

Ok, so my first day didn't exactly have me neck-deep in super-advanced code, but hey, it's the first day. I'm really looking forward to this job though, the boss is pretty cool and his politics are more in line with my own, and even more fun is the fact that I'm now labled "Senior Developer" and I'll be able to actually head up new projects and such.

That's it for me now. If you want more details, just gimme a call.