I've long agreed that many sites, like blogs, are better baked than\nfried. It makes for web hosting that's cheaper to run and simpler to\nmaintain. I've also often thought that using a database can be an\nanti-pattern for managing content. But, what I've also found is\nthat baked sites often yield a poor writing environment. That said, I think\nI'm going to give it another try, because I think I might have found a new\napproach that works for me.
\n\n\n\n\nAs I've mentioned before, I've flirted with a variety of\nplatforms for putting stuff from my brain on the web. But, the last time I\nswitched away from WordPress to Jekyll, I ended up switching right back again.
\nJekyll took way too long to generate my site and its 1150 posts, and I\ncouldn't figure out how to speed that up for previewing drafts without moving\nfiles around. I tried a few different external tools like\nMou and Marked, but the process\nnever clicked. I've also never quite gotten along with Ruby, so I didn't go\nfar with scratching my own itches on Jekyll.
\nOn the other hand, WordPress has a nicer writing experience. But, it's clunky\nin other ways. I'm always worried about all that PHP code sitting around\nfrying up page views, hoping no one figures out how to get at the publishing\nmachinery. I'm also less interested in hacking on PHP for fun, these days.
\nThe place where I've been having a lot of hacking fun over the past few years\nis in node.js. So, when I was thinking about trying static hosting for\nmy blog again, I started looking into node.js-based Jekyll clones.
\nBut then, it occurred to me that Gulp would be a fine tool for the job. In\na nutshell, like unix tools pipe character streams between tools, Gulp pipes\nstreams of files between small utility functions. All I had to do was build up\na small collection of file processing functions and glue them together.
\nAs it happens, someone else had already started that work for me! The\ncore of it, handling the posts, looks something like this:
\nfunction posts (path) {\n return gulp.src(path)\n .pipe(frontMatter({property: 'page', remove: true}))\n .pipe(taskUtils.filename2date())\n .pipe(marked())\n .pipe(taskUtils.summarize('\n\n<!--more-->\n\n'))\n .pipe(rename(taskUtils.postNameToDatePath))\n .pipe(taskUtils.applyTemplate('design/layouts/post.html'))\n .pipe(gulp.dest('build'));\n}\nPretty clean & straightforward, at least to my eyes.
\nStarting from Rioki's gulpfile.js, I hacked and iterated until I had\na gulpfile.js of my own, split into a directory of small task\nmodules. At this point, I've got a bunch of in-memory post indexes,\ndate & tag based archive pages, RSS feeds, and a handful of other templated\npages. I can push all the content to an Amazon S3 bucket with one\ncommand.
\nOh, and building the whole site only takes around 30 seconds. Still, that's\nnot fast enough for running previews while writing. So, I've broken things up so \nnew & draft posts lead to quick rebuilds when their files change - and I even trigger a LiveReload\nservice\nthat keeps a browser tab updated as I make changes in Vim.
\nAnd, best of all, I understand how the whole thing works. This stuff feels\nnicely maintainable and fun to expand in the future as a\nsaw-sharpening / yak-shaving opportunity. I might even take a shot\nat spinning off all the code from the content and release it as a standalone\nmodule installable from NPM in case anyone else wants to\ntry it out.
\nI found a WordPress-to-Jekyll exporter plugin. It\ngenerates a nice zip file download right from the site admin. That let me\ndump the 50 posts I've accumulated since the last switch.
\nAnd, a great thing about the YAML-and-markdown file format used by Jekyll is\nthat I was able to merge my posts from both decafbad.com and\nblog.lmorchard.com just by copying them into the same directory. So, I'm\nthinking that I'll revive my old blog by squashing it right on into the\nnew, and set up a handful of redirects to unify the whole mess.
\nHosting a static website on Amazon S3 is cheap and fast and low\nmaintenance. And, a module called gulp-awspublish can handle pushing this\nwhole site to S3 really easily.
\nTurns out I generate around 4750 files, between all the posts and tags and dates. \nIt takes about 30 minutes to upload the first time. But, gulp-awspublish\nkeeps track of MD5 hashes. So, next time I generate and upload, it skips all the\npages that haven't changed. That's just a handful of files, if\nall I'm doing is publishing one new post.
\nIt also seems like this module uploads one file at a time. I wonder if I might\nhack it to queue up a few dozen or so in parallel to speed things up? I doubt\nthat uploading thousands of files was the original use case, so it might do\nwith some tweaking.
\nI've got a simple template for this new blog, and I hope to keep it that way.\nBut, there's a lot of stuff in that sidebar. Well, I decided to tweak a few\nthings and suddenly I had 4750 files to upload to S3.
\nJust because the site is statically published doesn't mean some parts\ncan't be dynamic with the help of the client. Rather than put up wth\nregenerating & uploading all the things in the future, I yanked the sidebar\nout of almost every page and generated it as a separate resource.
\nThen, with a tiny bit of jQuery magic, I load that sidebar into\nthe page via AJAX. That shrank the size of the site overall, and it's so fast\nand cacheable that I never see any difference.
\nI think this will be one of the little keys to maintaining the site: Try to\nextract any common element used throughout the site, and push it into a\ndynamically loaded asset. Not everything can be done that way, but I think\nplenty can.
\nI'm also back to using Disqus for comments. They've got a great service, and\nthey're not a roach motel. They have a great API, and I even wrote a Python\nscript for decafbad.com that archives comments from closed\nthreads right into the blog post itself.
\nAt some point, I need to get that working again and maybe transliterate it\nover to node.js.
\nI've got some more I'd like to do with this stuff, but the main next steps are\nthese:
\nOf course, having skimmed through my posts over the years on this blog, I'd\nestimate about 25% of the whole thing is me grousing out loud about the long\nstretches I spend neglecting this place.
\nSo, who knows? Maybe you'll see my\nnext post show up sometime next June!
\n\n", "body": "I've long agreed that many sites, like blogs, are better [baked than\r\nfried][bake]. It makes for web hosting that's cheaper to run and simpler to\r\nmaintain. I've also often thought that [using a database can be an\r\nanti-pattern][antidatabase] for managing content. But, what I've also found is\r\nthat baked sites often yield a poor writing environment. That said, I think\r\nI'm going to give it another try, because I think I might have found a new\r\napproach that works for me.\r\n\r\n\r\n\r\n## From WordPress to Jekyll to WordPress\r\n\r\n[As I've mentioned before][platforms], I've flirted with a variety of\r\nplatforms for putting stuff from my brain on the web. But, the last time I\r\nswitched away from WordPress to Jekyll, I ended up switching right back again.\r\n\r\nJekyll [took *way* too long][jekyllslow] to generate my site and its 1150 posts, and I\r\ncouldn't figure out how to speed that up for previewing drafts without moving\r\nfiles around. I tried a few different external tools like\r\n[Mou](http://25.io/mou/) and [Marked](http://marked2app.com/), but the process\r\nnever clicked. I've also never quite gotten along with Ruby, so I didn't go\r\nfar with scratching my own itches on Jekyll.\r\n\r\n[jekyllslow]: http://blog.lmorchard.com/2012/06/16/blogging-like-a-blogger/\r\n\r\nOn the other hand, WordPress has a nicer writing experience. But, it's clunky\r\nin other ways. I'm always worried about all that PHP code sitting around\r\nfrying up page views, hoping no one figures out how to get at the publishing\r\nmachinery. I'm also less interested in hacking on PHP for fun, these days.\r\n\r\n## Gulp is great\r\n\r\nThe place where I've been having a lot of hacking fun over the past few years\r\nis in node.js. So, when I was thinking about trying static hosting for\r\nmy blog again, I started looking into node.js-based Jekyll clones.\r\n\r\nBut then, it occurred to me that [Gulp][] would be a fine tool for the job. In\r\na nutshell, like unix tools pipe character streams between tools, Gulp pipes\r\nstreams of files between small utility functions. All I had to do was build up\r\na small collection of file processing functions and glue them together. \r\n\r\n## Copying Rioki's homework\r\n\r\nAs it happens, [someone else had already started that work for me][rioki]! The\r\ncore of it, handling the posts, looks something [like this][tasks]:\r\n\r\n```javascript\r\nfunction posts (path) {\r\n return gulp.src(path)\r\n .pipe(frontMatter({property: 'page', remove: true}))\r\n .pipe(taskUtils.filename2date())\r\n .pipe(marked())\r\n .pipe(taskUtils.summarize(''))\r\n .pipe(rename(taskUtils.postNameToDatePath))\r\n .pipe(taskUtils.applyTemplate('design/layouts/post.html'))\r\n .pipe(gulp.dest('build'));\r\n}\r\n```\r\n\r\nPretty clean & straightforward, at least to my eyes.\r\n\r\nStarting from [Rioki's gulpfile.js][rioki2], I hacked and iterated until I had\r\na [gulpfile.js][] of my own, split into [a directory of small task\r\nmodules][tasks]. At this point, I've got a bunch of in-memory post indexes,\r\ndate & tag based archive pages, RSS feeds, and a handful of other templated\r\npages. I can push all the content to an Amazon S3 bucket [with one\r\ncommand][deploy]. \r\n\r\n[deploy]: https://github.com/lmorchard/blog.lmorchard.com/blob/master/gulpfile.js#L32\r\n[tasks]: https://github.com/lmorchard/blog.lmorchard.com/blob/master/lib/tasks/posts.js#L34\r\n[gulpfile.js]: https://github.com/lmorchard/blog.lmorchard.com/blob/master/gulpfile.js\r\n\r\nOh, and building the whole site only takes around 30 seconds. Still, that's\r\nnot fast enough for running previews while writing. So, I've broken things up so \r\n[new & draft posts lead to quick rebuilds when their files change](https://github.com/lmorchard/blog.lmorchard.com/blob/master/lib/tasks/posts.js#L13) - and I even [trigger a LiveReload\r\nservice](https://github.com/lmorchard/blog.lmorchard.com/blob/master/lib/tasks/posts.js#L29)\r\nthat keeps a browser tab updated as I make changes in Vim.\r\n\r\nAnd, best of all, I understand how the whole thing works. This stuff feels\r\nnicely maintainable and fun to expand in the future as a\r\n[saw-sharpening][sharpen] / yak-shaving opportunity. I might even take a shot\r\nat spinning off all the code from the content and release it as a standalone\r\nmodule installable from [NPM](http://npmjs.org) in case anyone else wants to\r\ntry it out.\r\n\r\n## From WordPress and Jekyll to Gulp\r\n\r\nI found [a WordPress-to-Jekyll exporter plugin][exporter]. It\r\ngenerates a nice zip file download right from the site admin. That let me\r\ndump the 50 posts I've accumulated since the last switch.\r\n\r\nAnd, a great thing about the YAML-and-markdown file format used by Jekyll is\r\nthat I was able to merge my posts from both decafbad.com and\r\nblog.lmorchard.com just by copying them into the same directory. So, I'm\r\nthinking that I'll revive my old blog by squashing it right on into the\r\nnew, and set up a handful of redirects to unify the whole mess.\r\n\r\n## Amazon S3 deployment\r\n\r\n[Hosting a static website on Amazon S3][static] is cheap and fast and low\r\nmaintenance. And, a module called [gulp-awspublish][] can handle pushing this\r\nwhole site to S3 [really easily][deploy].\r\n\r\n[static]: http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html\r\n\r\nTurns out I generate around 4750 files, between all the posts and tags and dates. \r\nIt takes about 30 minutes to upload the first time. But, [gulp-awspublish][]\r\nkeeps track of MD5 hashes. So, next time I generate and upload, it skips all the\r\npages that haven't changed. That's just a handful of files, if\r\nall I'm doing is publishing one new post.\r\n\r\nIt also seems like this module uploads one file at a time. I wonder if I might\r\nhack it to queue up a few dozen or so in parallel to speed things up? I doubt\r\nthat uploading thousands of files was the original use case, so it might do\r\nwith some tweaking.\r\n\r\n[gulp-awspublish]: https://github.com/pgherveou/gulp-awspublish\r\n\r\n## Page sections loaded via AJAX\r\n\r\nI've got a simple template for this new blog, and I hope to keep it that way.\r\nBut, there's a lot of stuff in that sidebar. Well, I decided to tweak a few\r\nthings and suddenly I had 4750 files to upload to S3.\r\n\r\nJust because the site is statically published doesn't mean some parts\r\ncan't be dynamic with the help of the client. Rather than put up wth\r\nregenerating & uploading all the things in the future, I yanked the sidebar\r\nout of almost every page and generated it as [a separate resource][sidebar].\r\n\r\nThen, with [a tiny bit of jQuery magic][sidebarajax], I load that sidebar into\r\nthe page via AJAX. That shrank the size of the site overall, and it's so fast\r\nand cacheable that I never see any difference.\r\n\r\nI think this will be one of the little keys to maintaining the site: Try to\r\nextract any common element used throughout the site, and push it into a\r\ndynamically loaded asset. Not everything can be done that way, but I think\r\nplenty can.\r\n\r\n[sidebar]: https://github.com/lmorchard/blog.lmorchard.com/blob/master/design/sidebar.html\r\n[sidebarajax]: https://github.com/lmorchard/blog.lmorchard.com/blob/master/design/js/main.js\r\n\r\n## Disqus and comment archival\r\n\r\nI'm also back to using Disqus for comments. They've got a great service, and\r\nthey're not a roach motel. They have a great API, and I even wrote a Python\r\nscript for decafbad.com that [archives comments from closed\r\nthreads][commentarchiver] right into the blog post itself.\r\n\r\n[commentarchiver]: https://github.com/lmorchard/blog.decafbad.com/blob/master/_bin/archive_disqus_comments.py\r\n\r\nAt some point, I need to get that working again and maybe transliterate it\r\nover to node.js. \r\n\r\n## Next steps\r\n\r\nI've got some more I'd like to do with this stuff, but the main next steps are\r\nthese:\r\n\r\n* Write more often\r\n* Write more consistently\r\n\r\nOf course, having skimmed through my posts over the years on this blog, I'd\r\nestimate about 25% of the whole thing is me grousing out loud about the long\r\nstretches I spend neglecting this place. \r\n\r\nSo, who knows? Maybe you'll see my\r\nnext post show up sometime next June!\r\n\r\n[bake]: http://www.aaronsw.com/weblog/000404\r\n[rioki]: http://www.rioki.org/2014/06/09/jekyll-to-gulp.html\r\n[antidatabase]: https://indiewebcamp.com/database-antipattern\r\n[exporter]: https://github.com/benbalter/wordpress-to-jekyll-exporter\r\n[platforms]: http://decafbad.com/blog/2011/06/08/moved-to-jekyll/\r\n\r\n[gulp]: http://gulpjs.com/\r\n[enthusiasm]: http://decafbad.com/blog/2006/05/26/confessions-of-a-serial-enthusiast\r\n[sharpen]: https://www.stephencovey.com/7habits/7habits-habit7.php\r\n[rioki2]: https://github.com/rioki/www.rioki.org/blob/master/gulpfile.js\r\n\r\n\r\n", "parentPath": "./content/posts/archives/2014", "path": "2014/10/20/static-blog-generation-with-gulp", "summary": "I've long agreed that many sites, like blogs, are better baked than\nfried. It makes for web hosting that's cheaper to run and simpler to\nmaintain. I've also often thought that using a database can be an\nanti-pattern for managing content. But, what I've also found is\nthat baked sites often yield a poor writing environment. That said, I think\nI'm going to give it another try, because I think I might have found a new\napproach that works for me.", "needsBuild": true, "prevPostPath": "2014/10/11/tootr-1", "prevPostTitle": "tootr: microblogging app, hosting not included", "nextPostPath": "2014/10/23/wtfomgbullets", "nextPostTitle": "What were all those bullets for?" }