First, what do I mean by a “team”? Well, a familiar example is a guild in World of Warcraft: A group of people with access to shared resources, that access controlled by permissions bundled into ranks assigned to each member of the group.

Guild management in WoW

While Django does come with Users, Groups, & Permissions, the concept of Roles within Groups to which Permissions can be attached seems missing.

If I try to extend the Django building blocks, maybe I could represent a “Team” as a bundle of Groups and treat those Groups as Roles? Or, maybe Groups should just become hierarchical – make it turtles all the way down. (I really like that expression, in case you can’t tell.)

With respect to that last point, I found django-hierarchical-auth. That seems like a good lead. But, I’m having trouble finding other projects using it. There’s feinCMS – I’ve never heard of that before, but that’s more my lack than theirs.

In the case of django-badger, I’d like to enable users to start teams, create badges that belong to the team, and allow other team members varying levels of control over those badges (e.g. creating, editing, awarding, approving nominations, etc).

So, there’s the matter of user interface – I’d like any user (not just site-wide admins) to be able to:

• create a team;
• manage team profile information;
• manage & grant roles based on a canned selection of permissions;
• invite other users as members, remove members;
• accept team invitations, request to join a team, remove oneself from a team.

The django-hierarchical-auth app comes with no views; it’s just a backend utility. So, if I grab that app, I’ll still have to build the UI and supporting models myself (i.e. for team profiles and permission sets).

So, that leaves me wondering if there’s some value in me building a reusable app atop django-hierarchical-auth that basically implements what that WoW guild management interface offers? Ultimately, that’s what I’m really hoping someone else has built for me.

Necessity is the Mother of Invention

The major focus of my workaday (and workanight) life last summer was the relaunch of the Mozilla Developer Network wiki.

It had been close to 18 months in the making, which usually spells death march and disaster. But, against many expectations, we did finally arrive at something for launch that neither fell over in flames immediately nor jettisoned a significant number of features with respect to what it was replacing. I think this is the first time in my career something like this went off with as few hitches as it did.

Your first questions might go something along the lines of, “18 months? For a wiki? Are you insane or just incompetent?” Well, it might help to note a few additional details:

• We call it a wiki, but it’s really more of a content management system that anyone can edit. It supports translation from English to 34 other languages, tracks revisions & content hierarchies, accepts file attachments, and sings & dances in a variety of other annoying-to-implement ways.
• At the time of switchover, we had over 50,000 documents to migrate with care from the old system to the new one. That body of content represents years of work, a non-trivial hunk of cruft and spam, and tickles a maddening array of edge cases.
• The wiki we replaced—i.e. MindTouch—supports server-side scripting in content with a language based on Lua—i.e. DekiScript.

Each of the above points represents its own special mix of horror and challenge, and I took on the bulk of the last two. That caused me a lot of stress, and I blogged a bit about that.

This blog post, however, focuses on that last point: KumaScript was built by backing into a semi-compatible replacement for DekiScript. That’s pretty much the worst way to go about building something, but it seems to have worked.

What Does It Look Like?

First, you might want to check out “Introduction to KumaScript” on MDN. It’s the best work in progress describing the ins-and-outs of the service. But for the sake of this blog post, consider this wiki document:

Here are three hellos:
{{ hello('3') }}

Now, consider this KumaScript template, named Template:hello:

<% for (var i = 0; i < $0; i++) { %>
Hello #<%= i %>
<% } %>

Put the two of these together, and you get this output:

Here are three hellos:
Hello #0
Hello #1
Hello #2

KumaScript on MDN consists mainly of the following:

• Templates implemented using Embedded JavaScript Templates
• Macros in wiki content that call templates with parameters
• Common JS inspired modules for reusable code used by templates

This quick introduction glosses over interesting things you can do with KumaScript—e.g. accessing data from external sources via HTTP, fetching content from other documents (also via HTTP). But, again, you can dive deeper by reading “Introduction to Kumascript” on MDN.

Scripting in Wikis

Why would one would even build such a thing as KumaScript? As it turns out, programmatically generating content is quite handy for composing documentation. Here are a few use cases:

• Localized macros for often-repeated constructs such as warnings, notes, tips, & callouts.
• Conditional content based on variables such as product, locale, & standards status.
• Transclusion of content, building documents from documents. (Try viewing the raw source of that page.)
• Mashups of data from MDN itself and from other sites and services like Bugzilla and Github. For example, here’s a self-updating Changelog of our code deployments on MDN. And, here’s the template behind that page.

It’s also worth pointing out that is different from scripting mixed with HTML like you get from ASP or PHP: There, you can process forms, personalize responses, and generally build web applications.

In the world of KumaScript, content scripting is a heavily cached thing and not tied to the current HTTP request. It mostly runs only when the document itself is edited, but can also be executed when we think dependencies or external data sources have changed.

In fact, KumaScript code doesn’t even have access to the incoming visitor’s request data at all—i.e. username, cookies, referrer header, et al—and instead we operate mainly on the content and metadata of documents.

Why JavaScript?

As I mentioned earlier, MindTouch provides for scripted content by way of the Lua-based DekiScript. It’s also interesting to note that the Wikimedia Foundation is working on a Lua-based scripting system for MediaWiki and Wikipedia. So, scripted content in a wiki isn’t an entirely crazy idea, in and of itself.

As for Lua, I think it’s a nice little language. It’s used in World of Warcraft and many other games. It’s known for being easily embedded into applications to grant scriptability. I can totally see why one would reach for it.

But, at Mozilla, we’re all about the web. The lingua franca of programming on the web is JavaScript. And, it doesn’t hurt that MDN already has a huge body of JavaScript documentation.

So, as far as harmonious language choices go, I can’t think of a better one for scripting content on MDN than JavaScript.

Boring Lets Me Sleep at Night

There are some exciting things about embedding a Lua interpreter into a wiki platform, as MindTouch and MediaWiki have done. Even having chosen JavaScript over Lua, I could have tried embedding a JS interpreter like V8 or SpiderMonkey into Python.

However, because I like to sleep at night and am not particularly clever about embedding languages within languages (yo dawg), I want nothing to do with this brand of excitement. Consider me a Hobbit among developers.

So, KumaScript is a standalone Node.js web service. That is, everything going into and coming out of KumaScript happens over HTTP. I understand HTTP a whole lot more than embedding language interpreters.

Don’t get me wrong: Node.js is an exciting piece of exotic matter in its own right. But, someone more clever than me maintains Node.js. And, I’m betting most of my co-workers and potential project contributors understand JavaScript, Node.js, and HTTP much better than embedding languages in other languages.

In fact, besides my overwrought glue code, KumaScript consists mainly of modules written & maintained by other people more clever than me. That’s even less work for me. One of the few things I like as much as sleeping at night is when other people fix bugs and build things for me.

HTTP ALL the Things

I really like HTTP. I’ve spent a good chunk of my adult life learning to understand it—so by this point it might be Stockholm Syndrome, but I think it embeds a lot of cleverness and useful decisions.

HTTP gives you interesting system boundaries. You can cache, scale, and abstract using intermediaries. There are nice identifiers (ie. URLs), status codes, and a rich arsenal of means to transport data and metadata (ie. methods, headers, and content types).

Having made KumaScript an HTTP service also means that someone other than MDN could use it. The interface was not built specifically for MDN, it’s neither dependent on Python nor Django. Fire up the processes and try running your web content through it—could be a wiki, could be a pile of static HTML. There is, of course, slightly more to it than that—but not much.

In fact, I really do hope someday someone beyond MDN tries using KumaScript.

Turtles All the Way Down

So, the MDN wiki—named Kuma, which means “bear” in Japanese—talks to KumaScript via HTTP. And, in turn, KumaScript talks to the wiki with HTTP.

In fact, although the KumaScript service itself is hidden behind a firewall, the wiki API used by KumaScript is open to the public. What’s good for the goose is good for the gander, after all.

The typical document rendering process goes something like this:

1. Kuma makes a GET request to KumaScript with the URL of a wiki document.
2. KumaScript makes a GET request to Kuma for the raw source of the wiki document.
3. KumaScript parses the source, looking for macros & inventorying templates.
4. KumaScript makes a GET request to Kuma for the source of each template needed.
5. KumaScript evaluates the macros by executing templates with the given parameters. This may kick off additional GET requests as needed by templates to load modules.
6. KumaScript responds to the initial request from Kuma with the results of macro evaluation in the document.

It’s turtles HTTP GET, all the way down. Well, except for when we want to do a preview before saving: In that case it’s an HTTP POST which kicks everything off at step #3, with raw source in the request body.

And though this might look like a Rube Goldberg machine, there are some nice qualities to all this HTTP GET traffic:

• Each GET is susceptible to caching, via the usual headers and semantics.
• Each GET can be serviced by a different process on a different machine.

I’m sure I could come up with more items after I have lunch, but this is just HTTP.

Content flows through request and response bodies. And—though this part might be a bit of a hack—I encode document context, errors, and messages using custom HTTP headers as a side-channel using the FireLogger protocol as an inspiration.

Security & Safety

Something I could have done with KumaScript was to simply allow wiki authors to drop hunks of executable code into the middle of documents. DekiScript seems to allow for this. But, we never really used it that way on MDN.

Instead, what we have is a system of templates and macros:

• Templates contain the JavaScript code in the form of Embedded JavaScript Templates. At present, these can be authored only by a core of trusted MDN editors.
• Macros call templates with parameters and dump the results of execution into the document. These can be used by anyone, and have a very constrained syntax.
• The content resulting from macro evaluation is sanitized such that it’s subject to the same constraints as hand-written markup.

Since KumaScript has no access to a user’s request data, we’re decently firewalled in terms of privacy and abusing the visitor. And, since the markup is filtered, it’s difficult to inject nasty XSS exploits and the like.

So, when I say security and safety, I’m thinking mainly about our servers: We want to sandbox this server-side JS such that it can’t abuse CPU, memory, or network resources. At present, my approach to this is anemic: Restrict code authoring to trusted people, and impose impatient timeouts on macro execution.

I have thoughts about improving this situation in the future, and hopefully expanding the ability to author JS templates. Because, remember, MDN is a wiki—anyone and everyone can edit it. I’d like that to include the JS code, if at all possible to do with relative safety.

Patches and pull requests are welcome, especially if you’re smarter than me about these things. (It’s not hard to be smarter than me about these things.)

Scaling & Stability

KumaScript scales like just about any web service. You can stick it behind a load balancer. Scale it horizontally by throwing more CPUs and processes at the problem. Cache the hell out of the responses. Throw a proxy in front of it to cache the hell out of outgoing requests to external services. Again, this is meant to be as boring as I can make it.

And, if a KumaScript process should happen to misbehave or starts having a seizure, just kill it and start another one. There should be no state to worry about, and the processes should start up really fast. Ideally, the logs will have recorded what went wrong and we end up with just a transient error.

Maturity & THE FUTURE

KumaScript turned a year old last month, has been in production since last summer, and had its last commit around 5 months ago. It has lots of tests, and it uses a version of Node.js from early 2012.

That doesn’t mean it’s abandoned, though: KumaScript is a mature project by most definitions. It’s been working well enough that I haven’t wanted to touch it. Most of the work goes on within the wiki itself, and KumaScript is meant to be the smallest piece it can be.

And maturity doesn’t mean I don’t have notions about future work. Off the top of my head, I’d like to get around things like the following:

• Improve template execution & sandboxing. Currently, if any one thing misbehaves in the document, the whole process gets aborted. Maybe instead, I should spin up a pool of processes: Each them can take care of executing a single macro, while a master process watches for CPU / RAM / network abuse and kills anything that behaves badly.
• Reconsider my possibly brain-dead approach to parsing source documents for macros using a PEG.js grammar. Or, maybe it’s good enough.
• Need much better error trapping and reporting throughout everywhere.
• Need much better use of statsd for measuring timings and suchlike.
• Maybe offer an HTTP proxy that runs all content through the service, for easier deployment atop existing sites beyond MDN.

And of course: Suggestions, patches, and pull requests are more than welcome!

I’m planning to return to primarily MDN work in a few months – but, right now, FirefoxOS is kind of a big deal.

Thus, like some of my other colleagues in the past month, I’ve gone from being a fully-operational battle coder on a project with which I have years of experience, to being a total noob with no idea how to find my way to “Hello world”. It’s been awhile since I last parachuted into the middle of an open source combat zone, so I fell back to my training to get my bearings:

1. Find the docs
2. Get the source
3. Run the project
4. Break something, see what happens, fix it
5. Fix a known bug, submit a patch
6. Try to improve the process of fixing things

Finding the docs

Oddly enough, MDN was a good place to start for docs. Though, there are FirefoxOS docs spread between MDN, GitHub, and wiki.mozilla.org. And, of course, they are each in various states of freshness and truthiness. Keep your wits about you; here be dragons.

Getting the source

Getting the source for FirefoxOS is a bit of a trick question – it depends on what you want to do, and how you want to contribute:

• mozilla-b2g/B2G is the mother of all repos. Only start there if you’re brave & interested in the whole shebang. Fair warning, though: When the smoke cleared hours later, and the README was done with me, I had acquired 17G of new stuff in my dev directory. But, you’ll have ALL the things, and you can flash FirefoxOS to compatible hardware.
• mozilla-b2g/gaia will be the most interesting if you’re like me and hoping to contribute as a webdev. That’s chock full of HTML, CSS, and JavaScript. It’s also pretty much free of pesky kernel code or device drivers that hurt my brain, because I’m not prepared to work at that low a level at this point in the morning.

There are other repos, for the Gonk and Gecko layers of FirefoxOS. But, for the sake of my sanity, I’m trying to avert my eyes for now.

Running the project

If you don’t have a compatible mobile device (and few do), you need a B2G Desktop build. While it’s true that many apps will run in Firefox Nightly, and there is an emulator you can build from the b2g repo, most apps are presently problematic in Firefox Nightly, and I’ve never gotten the emulator to build successfully.

You can either build your own B2G Desktop, or download a nightly build. I’ve done both, and generally I recommend sticking with the nightly Aurora builds.

On OS X, I was able to download a nightly build and launch it with a double-click. That got things running – hooray! But, it didn’t get me productive straight away: The nightly build comes equipped with its own embedded build of Gaia, which makes it handy for trying out your own 3rd party apps but not-so-handy for hacking on Gaia itself.

To improve this situation, you need to do two things:

1. Build your own Gaia profile from your own clone of mozilla-b2g/gaia. This is pretty much just running make or DEBUG=1 make from your git clone.
2. Run B2G Desktop from a terminal with command line options that tell it to use your new Gaia profile, rather than the built-in.

So, at this point, I have the Gaia source and a means by which to run it on my laptop.

Breaking things

After getting the project running as intended, I break something. And by that, I mean that I change a button title somewhere to “LOL BUTTS” or intentionally introduce an exception or syntax error somewhere. Then, I fix it. This is practice that exposes me to the guts of the thing; gets me into the edit/build/run cycle; shows me what it does when something goes wrong; and pushes me toward finding the tests.

But, I think I picked the wrong week to step into the arena, because everything came pre-broken. There were semi-known bugs preventing nightly builds of B2G Desktop from running my fresh clone of Gaia. It took me most of my first week to track down those bugs in bugzilla, find workarounds, and finally get something going. And then, it all broke again. And I found workarounds again.

Eventually, I was at a point where I built my own B2G Desktop from source with a particular patch from a particular bug applied, and that got me to the point where I could submit my first pull request to fix a broken test.

Happily, things seem to have stabilized since I started: B2G Desktop nightlies have been cooperating with bleeding-edge Gaia lately, so I’ve been able to stop building my own. So, now, when I find a combination of B2G Desktop and phone and Gaia checkout that works, I cling to that for dear life until it’s absolutely necessary for me to update one of the parts.

In short, FirefoxOS is very much in flux, right now. Bring a hard hat and expect delays.

Fixing things

Once I’ve broken something and then fixed it, I usually have a sense for how to get real work done. Since then, I’ve been slowly wrapping my head around the core Calendar app, and my bug fixes have finally started trickling in.

Cobbling together a reliable Gaia-hacking workflow is complicated, though:

• Some Gaia apps work in Firefox Nightly, at least partially. This is great, because Firefox has all the great tools like the DOM inspector and web console. And, for the most part, you can just save-and-refresh to see changes.
• All Gaia apps work in B2G Desktop, usually, though sometimes there are… quirks. This is good for seeing how changes work in a closer-to-real environment. But, B2G Desktop has none of the web developer tools offered by Firefox. And, to see changes, I generally have to save, relaunch B2G Desktop, then relaunch the app within the simulator. That’s pretty inconvenient.
• All Gaia apps work on the Mozilla-supplied dev phone – which almost no one has. Well, they work more often than they do on B2G Desktop and Firefox, anyway. But, including a phone in the dev iteration loop is downright painful: As opposed to the usual webdev experience of save-and-refresh, the phone demands a save-build-flash-restart-relaunch cycle.
• There is one awesome thing, at least for the Calendar app: There are unit tests which run in B2G Desktop, but are driven by a node.js controller in my shell. That lets me beat up on my code & logic in a “headless” context before jumping into Firefox, B2G Desktop, or the phone for a fuller integration test and a manual run-through. In fact, the unit testing framework even watches for file changes and will re-run tests right after a save, complete with a Growl notification of pass/fail.

Room for improvement

Slowly but surely, I’ve gotten my minigun barrel spun up to start firing piles of lead bullets at Gaia bugs. But, it has been and continues to be a challenge. I’m hard-pressed to recommend it as a fun-time leisure project to anyone without a Mozilla paycheck, or even as an off-hours interest for those who live-and-breathe Firefox. This is to be expected, since it’s not even a v1.0 project yet – but, I really, really hope things improve in the not-so-distant future.

• We need to make sure that new contributors can get quickly from cloning the source to running the code. And it should always work – as in big, noisy automated tests fail when it breaks. I don’t care if it works fine on the phone if I can’t hack on it, at least from a contributor perspective.
• We need a consistent and convenient development environment for Gaia. All day, I bounce between things that are partly broken in Firefox, partly broken in B2G Desktop, and mostly working on the phone. This kills productivity and enthusiasm, all day. And, many times, no one else in the #gaia channel on IRC knows what I’m on about when I say something’s broken – because everyone’s dev environment is a unique little snowflake.
• My ideal workflow would never leave Firefox and my editor: Consider it a variant of the Responsive Design View, and ensure all the various APIs expected on a real phone are either working properly, polyfilled, or usefully stubbed out. Then, when it’s perfect in my most comfy environment, I can try it on a phone or standalone simulator as an afterthought. The upcoming built-in Firefox OS Simulator (née r2d2b2g) looks promising, but it needs to be seamless.

Still digging

Even with the challenges, and even if my contributions are small, I’m happy to have the opportunity to work on FirefoxOS. It’s a big deal; it’s the next Firefox. So, I plan to keep ramming my head against this stuff to help make it better, and I hope we can start taking time to make making it better better, too!

Assumed rights

This started out as a comment in response to Sorry Facebook, This Was A Privacy Bungle! Here’s What You Should’ve Done, over on ReadWriteWeb. The post wraps up like so:

So this isn’t the fault of Facebook’s users. We knew what we were doing then, when we posted to our Walls. And we know what we’re doing now. The lesson here is that Facebook should have given us the option of selecting the privacy setting for those old Wall posts. Or maybe even made those old posts available to “Close Friends” only, as the default. Then we could adjust if we wished.

Facebook’s mistake was that it had no right to assume that our “Friends” of 2007-09 means the same thing as “Friends” in 2012.

Sorry, but no. The mistake was that you could assume many rights over content you’d submitted into Facebook’s ownership. If you had assumptions of rights or ownership, then in fact you did not know what you were doing then when you posted to your Walls, then or now.

Darth Zuckerberg?

Maybe this whole thing seems a bit Vader-ish. (ie. “I am altering the deal. Pray I don’t alter it any further.“)

But, even if your impression was that what you posted back then was “just between us” – the fact is that Facebook has always been well within their rights to alter the definition of “us” at any point they liked. I’d be amazed if the terms of service (which no one reads, of course) didn’t give them plenty of room to do things just like this.

Of course, I’m not a lawyer. But, I can imagine there are things companies put into terms of service that run afoul of actual laws. But, Facebook has lots of money, and that yields lots of incentive to look for legal exploits.

And, accordingly, people have sued and won where Facebook was vulnerable. But, I’m reasonably convinced that voluntarily posting your secrets to Facebook doesn’t come with a legal guarantee of privacy. Maybe a good lawyer (ie. better than those at Facebook) could prove me wrong on this. But otherwise… umm… don’t do that.

It’s not your backyard

You don’t own the spaces you inhabit on Facebook. You’re enjoying a party at someone’s house, and you barely know the guy. In fact, your content is the currency that pays for the booze (ie. the privilege of using their servers). That’s why it’s free-as-in-beer: You’ve given them what you post, instead of money. That’s valuable stuff, if they can ever quite figure out how to sell it.

You and your friends can tell yourselves that your little clique on the quiet patio is a private gathering. But, that’s still not your house. Don’t be surprised if gossip gets overheard.

Richard MacManus wrote that “[Facebook] has unilaterally decided to impose a new concept of privacy onto its users.” But, that’s like saying the homeowner unilaterally decided to put out the bonfire in the backyard. One might ask the guests’ opinion about the bonfire – but that’s just being a gracious host.

A better scenario might be if the homeowner decided to start filming a reality show. Still, I think you agreed to a release at the door. It might have even been printed the plastic cups, and drinking the beer sealed the deal. That release covered this contingency, no matter how intuitively distasteful it seems. (Sounds shady to me, but then so do terms of service pages.)

The cake: eating vs having it

You can’t both pay to party by supplying your content and then later claim ownership of the payment. You ate the cake; you can’t take it home after. You can express your feelings of stomach ache (which is also content), but the lesson here is Facebook never was and never will be a private space. There might have been perceived partitions at one point, but those partitions can move whenever the homeowners want.

Where to go from here

As I see it, you have three options:

1. Pay to create your own private social spaces on the web. That, at least, makes you the customer.
2. Accept Facebook for what it is: A party in a stranger’s backyard, at which you are both a guest and the product on sale.
3. Lobby your lawmakers to outlaw what Facebook does with your content. Good luck.

I’m willing to bet #1 and #2 are cheaper and much more practical than #3.

Fractal of choice

To sum up what I wrote earlier, there’s a spectrum of what choices you make and what choices you leave up to someone else:

• You might choose to live entirely within the Apple ecosystem, relying on expert designers to provide elegant and beautiful experiences.
• You might decide to buy a PC laptop with Ubuntu pre-installed and embrace the gritty world of open source software.
• You might build a PC desktop from parts ordered ala carte—maybe even compile your OS and all applications from source, Gentoo or FreeBSD style.
• You might decide to construct an 8-bit CPU out of ICs and a big mess of wires, and cobble together your own port of Microsoft BASIC.
• You could gather piles of various exotic materials and bake your own semiconductors and ICs at home—but really, who does that?

There are many choices to make, and it’s even your choice to decide at which fractal level of complexity you’re most comfortable. So far, so good.

Changing your mind

But, what about after you’ve made your choices about choices—can you change your mind later?

• Can you take your tools with you to a new ecosystem?
• If you can’t take them with, can you find alternatives?
• Are your thoughts and work expressed in ways that leave you free to change tools?
• Can you take possession of the thoughts you’ve poured into a tool or service?
• What happens when your relied-upon experts go out of business?
• Or, better yet, what happens when they cash out to Google or Facebook or Yahoo?
• Do you own the device in your hand? Or, have you just gotten a limited license to use it?

And, this is probably overly clever, but when I write “change your mind” I mean both:

1. reconsidering your decisions and
2. altering your cognitive system.

Because, as I’ve written before, these things are cognitive prosthetics—they’re a part of your thinking apparatus. Again, it goes back to how much time you’re willing to devote to sharpening your saw with intentional use of technology.

Personally, I resist the notion of allowing parts of my extended mind to be controlled more by third-party experts than me. Even if I choose for now to leave myself in their hands, I want clearly marked exits and an escape plan.

Intersection of axes

So, if freedom {of,from} choice and freedom to change my mind are orthogonal, what does that look like? (Maybe I’ll come back to this post with a drawing.)

I would think that a high degree of freedom of choice would leave you quite free to change your mind later. But, I could imagine reworking that Big Mess ‘o Wires would be a huge pain. So, I probably wouldn’t use a BMOW as my daily workstation.

On the other hand, some of the best designed systems are trending toward hoods welded shut. Shouldn’t it be possible to start from a product of expert design (an iPhone, for instance), only later to pop the cover off (literally or virtually) and do your own thing? There are jailbreaks—but why are you in a jail? And, doesn’t it seem like they’d make it airtight if they could?

Is it just that app pirates are ruining it for the rest of us? I mean, it’s mighty convenient to have your customers locked in the trunk: It’s harder for them to hurt themselves or the upholstery, and they can’t easily wander off if you make some turns they don’t like. But, that’s tinfoil-hat territory, and I could rant all day.

Android seems to be doing well, and it’s mostly open. I’ve switched device vendors a few times, and all my apps have survived the changes. I’ve flashed my own choice of firmware on a few of my phones, and have been pleased. I have a sense of deeper choices available, should I ever want to chase after them.

Palm’s webOS devices were crazy open—not open source per se, but you could unlock root with the Konami code and apply patches to core applications. Alas, webOS crashed and burned at HP. My hunch is that being open isn’t what did them in, though.

Mozilla (disclaimer: my employer) is headed into that webOS territory with FirefoxOS. In that currently-coalescing ecosystem, the apps come from the open web, aren’t gate-kept by a single marketplace, and can even run on non-FirefoxOS devices. I suspect it won’t be as slick an ecosystem as Apple’s, because the loosely-coupled gates will let in more dirt. But, it will be high on the freedom-to-change axis, and the quality of freedom-from-choice options will limited only by the talent of the designers involved.

Conclusion

So, I didn’t really mean to write an advertisement for FirefoxOS—I work for Mozilla because I love the mission, I’m not trying to shill. Really, I just think it’s very important to have a sense of your choices and freedoms when you accept technology into your life. And, of course, I have some very strong opinions, but this post and the one before it are intended as more about choice in general than my own choices or angst. If you’re happy with your choices, even upon close examination, then that’s great—just keep your wits about you.

Tasting menus and wine pairings

My wife and I like to go out a few times a year on date nights to nice restaurants. These restaurants have tasting menus with wine pairings. This is dinner on cruise control, and we let the chef drive. It’s great when it works, because the chef makes expert choices.

Occasionally, it doesn’t work. There might be something we don’t like: Rabbit are pets, for instance. We’re not fans of seafood, though some menus have surprised us. We also have food allergies. Usually—though not always—we can ask for a substitution. Otherwise, we chalk it up to adventure.

But, overall, ordering from the tasting menu and trying the wine pairings makes date night simple with a side of novelty. It’s on the high end of freedom from choice. We choose to go out, and we choose the restaurant, but we leave the rest to experts.

Suds and swine

On the other end of things, I like to make homebrew—zymurgy, not software. There’s a spectrum of choice to be had here, too. But, even with a kit, the process is much more involved than just ordering a pint. I also have friends who are great cooks. When we all get together, it’s a party.

But, that party is about the food and drink, about sharing the process and the product. It’s not a simple date night: The choices are the focus as much as the participants. It’s a chance to show off and try new things, become experts ourselves.

Power and ease of use

Where am I going with this? Well, Dave Winer said this in a podcast:

In software, the tricky parts are where the power comes from, and the ease of use comes from where you limit the amount of options. The more limited the options, the easier it is, but the less powerful it is. It’s just an engineering tradeoff there. It’s like the tradeoff between heat and motion in physics.

The folks at Apple are masters of tasting menus and wine pairings when it comes to consumer electronics. You pick Apple because you don’t want to think about most of the choices. They have experts in various forms of design, and their products are elegant, beautiful, and generally work great together. It’s blindingly obvious from their bank balance that this is something people are more than happy to pay for.

Why? Because you have better things to do with your time than to futz with impedance mismatches and arcane interfaces. You know, like do the things for which you bought the gadgets in the first place (eg. work, play, communicate, etc).

On the other hand, you have something like the Linux ecosystem. Therein, many others have made choices for you. But, all the choices are still open to you—and many must be made by you. The result is kind of a mess at times, with little or no top-down unifying vision. But, the grit and the loose coupling between the parts are the stuff of freedom. Freedom of choice is dirty and complex, especially when lots of players are involved.

Choose not to decide, or DIY

Here’s the thing: If you choose not to decide, you still have made a choice—you’ve just chosen to let an expert make most of the choices for you.

What happens when disagree with some of those choices? If you’re lucky, you can ask for substitutions (or tweak some options). But, you might have to go to a different restaurant (ie. another manufacturer). If you’re really unlucky, you’ll find that you disagree with almost all of the restaurants and manufacturers. Sometimes, we just grin and bear it, because switching is too inconvenient.

And beyond switching, do-it-yourself is just plain hard. You can crawl up the spectrum toward greater freedom-of-choice—but that demands greater knowledge, skill, and attention. You have to discover the choices, make them, and then track what you’ve done. In fact, it rises to the level of a hobby at a certain point, and you spend as much time (or more) on the infrastructure as the things it’s there to support.

Sharpen the saw

Here, I guess, it’s a personal decision: How much time are you willing to spend on the devices through which you do your thinking? Because, that’s what computers, tablets, smartphones, and software are: They’re cognitive prosthetics, addons to your brain that expand your capabilities to think, remember, and communicate. How comfortable are you with the choices made by your chosen experts in shaping the patterns of your mind?

Me, I’ve been a hobbyist in this field since very shortly after I learned how to read. My take on the power / ease-of-use spectrum was best put by Doug Engelbart, who asked “Would you rather ride a tricycle or a bicycle to get across town?” I’d rather have the freedom to program and recombine (and crash and twist my ankle), than take on an extra stabilizing wheel or slow down.

But, even if you’re not willing to rise to the level of cognitive prosthetics hobbyist, think about sharpening the saw. Have you ever tried meditation? Written a journal? Taken training? Practiced a skill? Have you ever taken a critical look at the technology in your life and considered how it’s serving you and your mind? It’s all in the same category, to me. It’s an effort, and it takes time—but it’s self-investment.

Food for thought

In my head, earlier versions of this were more judgmental: Holy crap how can so many people be so passive about the technology in their lives?

Of course, it’s obvious: People don’t have time for this crap. You’re the weirdo who’s been doing it almost since birth.

Still, just like attention to diet and exercise (of which I need more, myself), I think intentional use of technology is crucial. Take stock of your choices and freedoms. Be mindful of where you are on the freedom-of / freedom-from spectrum.

Do your gadgets and services make you more free and capable in ways about which you care? Have you given up things you’d rather not have given up (eg. time, cost, privacy, flexibility, lock-in, reliability, longevity)? Do you trust what your chosen experts are doing for you, are they working for your interests?

Of course, in my personal life, it’s a hobby. And it’s my job. And I spend way more time on mindful technology than I should, versus mindful eating and living. Like I said, I need to work on that. But, hopefully, I’ve managed to lay out some ideas in this post about freedom, choice, and technology without coming off as a total angry nerd.

So, I think I’m going to build a simple app and blog about it. And, these days, the first thing I think about when starting a web app is: How do I get it onto the web?

Start on the web

Okay, so getting on the web is not the first thing. The very first thing is to create a new github.com repo (complete with README.md and .gitignore) and then:

cd ~/Dropbox/Public/
git clone git@github.com:lmorchard/my-project.git
cd my-project

And then, I start thinking about getting it onto the web.

That might sound backward: If all I have is a README, aren’t I jumping the gun? Well, no—I like to iterate, especially when experimenting with something new. Start simple, try one little thing, repeat. And, because this is a new pan-device world we’re living in, I’m often sitting on my couch hacking with a laptop in front, tablet to the left, and phone to the right.

So, before I even write the first line of “Hello world” in HTML, I want to have a way to get at it from all the gadgets. If it were just my laptop, I could open a file:// URL and be done with it. Oh, but not really: Many of the new HTML5-and-friends technologies depend on things like domain origins and real HTTP connections.

I could start up a disposable HTTP server in the project directory on my laptop and point the other gadgets at that. That works, to some extent. But, that’s limited to my LAN, I can’t share it with others for quick alpha testing, and I can’t check it out when I’m away from home and the laptop is closed.

That leaves me with my own web server. But, you know, that’s a pain in the butt and I’m getting tired of being sysadmin to my own server. Let’s take it to the cloud!

How I (ab)use Github Pages

Since I started with GitHub, my project is already on the web. Or, mostly it is, but just not exactly how I’d like it. To get the project published at a URL, this is what I do:

git co -b gh-pages
git push origin gh-pages
git co master

Give it a few, and you’ll receive an email from Github Pages (re: [my-project] Page build successful). Give it a few more, since this is the first time for the project. Shortly, the README shows up at an URL like the following:

http://lmorchard.github.com/my-project/README.md

After that, whenever I want to publish updates to the app, this is what I do:

git co gh-pages
git rebase master
git push origin gh-pages
git co master

And, within seconds (usually), my app is updated on the web. With free hosting, because the folks at GitHub are really nice.

But, though this works nicely for an app demo or beta, try not to abuse their hospitality by using your GitHub Pages URL as the official distribution point for your app.

How I (ab)use Dropbox

You know, all that committing and pushing and rebasing can really get tiresome. And, it craps up my commit history while I’m just kind of twiddling some bits back and forth to see what happens. We’re living in the future, damn it, I don’t have to put up with this!

And, so, I don’t. Maybe you noticed that I ran git clone from inside my Dropbox Public folder. Anything in that folder is already on the web: Each file has a public URL, and I can acquire it with a right-click in Finder (ie. Dropbox > Copy Public Link). The result is something like this:

https://dl.dropbox.com/u/2798055/my-project/README.md

It’s a crufty URL, but an URL nonetheless. Dropbox public links also mirror the folder structure, so that an index.html can load all the CSS and JS it wants from relative URLs just like a normal web host.

With Dropbox, then, my hacking cycle with Dropbox looks like this:

1. Save in MacVim
2. Switch to Firefox
3. Refresh
4. Repeat

More often than not, my app has gotten updated on the Dropbox servers somewhere between steps #1 and #3. That’s a much tighter iteration loop than doing the GitHub dance. In fact, it’s pretty much indistinguishable from just working on my laptop with file:// URLs or a disposable HTTP server on my LAN.

There’s some bad news, though. I’ve just read that new accounts created after 7/31/2012 no longer get a Public folder. So, it would seem that the days of carefree hosting-by-Public-folder are on the way out. But fret not: the folks at Dropbox have provided an alternative for hosting static web apps under development. They’ve also opened up their API for use by open web apps with dropbox-js. In fact, this API will play a part in the app about which I plan to blog.

How I use Amazon S3

So, (ab)using GitHub and Dropbox for hosting your app under development is fine and dandy. But, you definitely don’t want to rely on them for publishing your app for wider distribution. There ain’t no such thing as a free lunch (TANSTAAFL), and someone’s bound to shut you down or send you a bill—best to get that out of the way up front.

As it turns out, Amazon Web Services will happily accept the change from your penny jar to serve up small collections of web content. Well, they will after you’ve turned the physical pennies into numbers in a bank account, but my point is that Amazon S3 is super cheap. And, since I’m talking about a single-page web app here, plain old static web hosting is all you need for now.

In fact, if you own a domain name, you can control the URL from which S3 serves up your content. Basically, you just:

1. create a DNS record with a CNAME to S3 (eg. my-project.lmorchard.com -> s3-website-us-east-1.amazonaws.com);
2. create a bucket named for your domain (eg. my-project.lmorchard.com);
3. check a box in the properties panel to enable static hosting.

The above is vaguely more advanced than either GitHub or Dropbox, but this is playing with power. And, with great power comes great responsibility to read the fine manual. Still, it’s not that hard and it’s not that spendy.

How I make Amazon S3 work like Dropbox

So, great: You’ve got Amazon S3 serving content at a URL of your choice. And, you can upload content using the control panel or one of the many fine desktop clients available. But, that sounds a bit like the GitHub deployment dance, just GUI-er. Not bad for Official Releases of your app—but what a pain for development!

Apropos of that, I have a small trick on OS X that makes Amazon S3 as convenient as Dropbox:

• Install kicker;
• Install s3cmd and configure it with your S3 credentials;
• Run this from your project directory in a terminal:

  kicker -c -e ‘s3cmd -vfrP –exclude=”swp” –exclude=”.git” sync . s3://my-project.lmorchard.com/’

What this does is start kicker to monitor your files. Whenever anything changes, s3cmd uploads the changes to Amazon S3. After the first run, my content shows up at an URL like:

http://my-project.lmorchard.com/README.md

So, again, my hacking cycle with Amazon S3 looks like this:

1. Save changes
2. Switch to browser
3. Refresh
4. Repeat

Again, this turns out to be nearly indistinguishable from working on localhost or with file:// URLs, because the kicker-s3cmd team has generally already shipped off my changes before I can get to the browser. I’m sure something similar to the above can be cobbled together on Linux or Windows, but I don’t work there so I’ve not bothered to figure it out.

In conclusion

Developing open web apps on the open web can be fun, fast, cheap, and convenient. I’ve got more posts forthcoming, at least in my head, but hopefully this one gives some starting points.

I may also revise and further develop this post, since I left quite a bit of reading to the reader with links to follow. There are also even more cheap and easy web hosts, so maybe I’ll explore those as well. In the meantime, feel free to leave some comments! ]]> http://blog.lmorchard.com/2012/09/18/first-get-it-onto-the-web/feed 2 http://blog.lmorchard.com/2012/09/17/on-the-other-end-of-a-self-imposed-death-march-project http://blog.lmorchard.com/2012/09/17/on-the-other-end-of-a-self-imposed-death-march-project#comments Mon, 17 Sep 2012 20:25:03 +0000 lmorchard http://blog.lmorchard.com/?p=213 So, I’m a couple of weeks back from a well-deserved vacation taken after the launch of a project well over 18 months in the making. I kind of overworked myself, voluntarily, and that needs to change.

What project? Kuma on developer.mozilla.org. What’s that? Well, it’s generally described as a wiki, but though I think we’ve gone a long way away from “The simplest online database that could possibly work“. At this point, it’s a content management system, complete with revision management, WYSIWYG, collaboration, community, and localization features. It’s a wiki insofar as anyone can edit it, but Kuma’s not exactly simple.

The parts I had the heaviest hand in were:

• Content migration from the previous incarnation, which employed the MindTouch platform to a very full extent;
• Development of KumaScript, a server-side JS platform for dynamic content heavily inspired by (ie. cribbed entirely from) DekiScript.

As it turns out, those were two of the hardest parts of the project. No, I wasn’t alone: I’m a member of a team, and my teammates did plenty of work themselves. But, I’ve found that I have a lot in common with Mordin Solus, a fact that made me actually cry during this scene:

(Spoiler alert: Mordin gets blown up.)

That’s why I say this was a self-imposed death march. No one nailed me to my desk, and no one on my team was a slacker. In fact, although there was a lot of pressure to get this project out the door ASAP, managers and others quite often told me to take it easier. And furthermore, working on myself to ease up on this particularly neurotic sentiment was a major theme of recent performance reviews.

So, to be clear: This is about me, and not my team. My team is awesome.

That said, my motivations for the work on MDN are pretty simple:

• I think producing tech books online, collaboratively, is the future;
• I love Mozilla and the open web.

And, so it seems, Mozilla loves me back. I got atta boys all around, from managers and awesome teammates alike. So, my efforts certainly didn’t go unappreciated. In my experience Mozilla is a brilliant place to work and is an organization that appreciates both smart and hard work in ways both conventional and unexpected.

All that said, though, I really do need to ease up. There are plenty of things in life at large that I’ve been neglecting lately – and that’s not good: Those are the things that make my work possible to begin with.

But, I’m not burnt out and I’m not moving on. I didn’t bust my ass on MDN just to wander off from it, and I can’t think of a better place to be than at Mozilla (especially right now). But, it’s time to take a bit of a more human pace with things. Contribute, take breaks, trust my awesome teammates can kick as much ass as I demand from myself.

Because the MDN team totally does kick so much ass.

There’s still the sysadmin-as-remote-pair-of-hands-driven-by-trouble-tickets thing going on, but less so every day. More and more, we’re getting into continuous deployment. Our sysadmins are really smart cookies: They’re steadily replacing themselves with scripts of varying sizes. That way, they can get on to more interesting things. In fact, just recently, they gave us a great Big Red Button that deploys code from a git repository to the live site on demand. And, for some reason, I’m one of the people who got a key to it.

Okay, maybe not that exact button, but ours is very keen. Of course, when it broke once, I felt like the Pakled on the right (and not Geordi):

But, the Big Red Button hasn’t broken very often. So, I’d rather be a Pakled some of the time than pretend to be a rocket scientist all of the time.

Oh yeah, and we’re using GitHub a lot these days. It’s crazypants awesome for collaboration, and it fits right in with our style because we want to do everything in the open anyway. We’ve even somehow lured people who don’t normally code into submitting Pull Requests. And, it’s amazingly powerful to refer to specific lines of code and commits by URL in IRC or email.

I’ve also been getting a lot better at building development environments, using Puppet and Vagrant and Amazon EC2. I haven’t quite gotten to the point where I can borrow IT’s Puppet manifests directly – ie. those scripts with which sysadmins are replacing themselves.

But, we are maintaining the recipes for spinning up dev servers right alongside the code. Our READMEs are shrinking, rather than growing – and yet we’re still adding more services and dependencies. I no longer really fear totally jack-knifing my dev rig, because I can just toss it and conjure up another in the time it takes me to run to lunch.

Let’s see, what else? I’m sure I’ll think of something. Until then, SHIP IT. (The post, I mean.)

So, I build my own replicas of production sites, as best as I can. I deploy exploratory rovers into those environments, and try to account for what could happen based on everything I know up to this point. Once I’ve done all I can to prepare, I fire up the comms dish and beam out a request to the distant probe. This takes time because I have to wait until the receiver is in alignment, and there’s a transmission & reply delay due to distance. (In other words, I file a bug and wait for someone to pick it up. That takes awhile, because we’re all busy people, and I’m a telecommuter who can’t just go tap someone on the shoulder.)

I cross my fingers and hope I’ve built up a good-enough model of the remote environment, and maybe I’ll find that my hypothesis will be validated. If not, well, then I at least hope I haven’t broken anything, and hope that there’s new information in the reply that I can assimilate into my replica environment here back at home for an improved next attempt.

But, I’m not entirely sure how to improve this situation. If I had direct access to production servers, I’d be able to do more. However, if I had that access, I’d be expected to do more. You know, like wear a pager and wake up at 4am to fix an alarm condition. I’ve done that job in the past, and I don’t envy anyone who has it now.

There are also, I imagine, trust and confidentiality issues: even though I write the code, I’m not necessarily qualified to be in contact with certain kinds of sensitive data on a daily basis. (And by not qualified, I mean not bound by a certain class of contractual or legal obligations and not subject to certain kinds of training. I’ll say you can trust me, but trust is sometimes a formalized thing.)

Still, some days I like to imagine that what I’m doing is trying to debug the behavior of a robot in orbit around Mars. It keeps me from putting too many dents into my desk with my forehead.