Tuesday 17th November, 2009
I’m posting this to announce a project I’ve been working on for a few weeks now; something which has only just reached a state of relative stability and functionality. It started off slow, on October the 1st (it was a birthday present to myself), but I’ve since spent a lot of time tweaking, extending, refactoring and cleaning. I now think it’s ready for a public preview.
Markdoc is “a lightweight Markdown-based wiki system.” I started work on it because I was slightly fed up with the state of documentation tools out there. I had a choice between:
MediaWiki, which needs a whole SQL database and a PHP web server just to run.
API docs (via epydoc or otherwise), which would auto-generate documentation from the docstrings in my source code. Fine, unless I want to document something other than functions, modules and classes.
Sphinx, which is a great project, but unfortunately uses reStructuredText, which I just can’t enjoy using; it’s also geared almost completely towards documenting software.
Something completely different: I could make something myself, which would provide exactly what I need, at a significant time cost.
Being the hacker (and perhaps masochist) that I am, I went for the last option. It was clear that the new system would have to meet several criteria:
VCS-friendly — it would have to be entirely plain-text, ruling out anything based on SQL. It should be trivial to diff and merge files, and even whole wikis.
VCS-agnostic — it shouldn’t make a difference if I’m using Mercurial, Git, SVN or indeed nothing at all: the wiki shouldn’t care about how I’m managing its files.
Plain-text — it’s no use having a wiki which uses HTML, XML, binary or any other unreadable format. I should be able to read the wiki without even knowing that it’s anything more than a collection of plain-text documents.
Statically-generated — A wiki with a web interface would be nice, but it would be a million times more useful if wikis could compile down to standalone, static HTML directories, for later viewing or tarballing.
Cross-platform — the command-line interface (CLI) to the wiki should run on every platform it possibly can; that’s just good programming practice. I don’t want to go out of my way to make it work on every single platform, but platform-agnosticism is a fair enough criterion, and gives a certain robustness to the software.
Continuous — it should be possible to serve the wiki via HTTP in one process, and in another to update the source and re-build the HTML, without having to restart the HTTP server. This could be used in a situation where a wiki is being served from a DVCS node, and a post-commit hook re-builds the HTML when changes are pushed to it (N.B. I have actually done this with Markdoc, and it’s insanely simple).
Markdown-based — Markdown is my favorite plain-text formatting syntax, and the Python implementation is fast, correct and extensible. It already has Pygments syntax highlighting, definition lists, TOC generation and more. The wiki might as well exploit these features.
Flexible — hackability is important in any piece of software, but I want to be able to create wikis for many different scenarios (software documentation, static website generation, blogs, et cetera), without sacrificing simplicity.
Seems like a difficult set of constraints to meet, doesn’t it? Well, it did take me a month and a half to complete.
It’s pretty simple, really. For those familiar with Python’s installation routine, this is the usual:
$ hg clone http://bitbucket.org/zacharyvoase/markdoc
$ cd markdoc/
$ pip install -r REQUIREMENTS
$ python setup.py install
That’s it for installation. You’ll need pip, which
is a next-generation package management tool for Python. You can get it with
easy_install -U pip. If your OS has a package manager, it may be available
there too.
You’ll also need Python 2.4 or above (2.5+ is highly recommended), and you must
have a working rsync binary available on your search path (rsync comes
out-of-the-box with most major OSes today, including Mac OS X and Ubuntu).
Please note that I am working on streamlining the installation process; it’s just not my main priority whilst Markdoc is under heavy development.
This is also pretty simple.
markdoc init creates the wiki skeleton; this is just three directories and a
single file, all empty to begin with.
$ markdoc init my-wiki
# ...logging output...
$ ls -A my-wiki/
.templates/
markdoc.yaml
static/
wiki/
Enter your wiki and start editing files in the wiki/ sub-directory. Wiki pages
must end in one of [.md, .mdown, .markdown, .wiki, .text]; this may be
configured by editing the wiki’s markdoc.yaml file.
$ cd my-wiki/
$ vim wiki/somefile.md
# ...edit some markdown files...
After you’ve edited enough, and you want to see your files rendered, just run
markdoc build in the wiki root.
$ markdoc build
# ...more logging output...
That will build your files into .html/, which should now contain a number of
HTML files corresponding to all the pages you wrote. You can now serve up these
files with markdoc serve:
$ markdoc serve
# ...even more logging output...
Now visit http://localhost:8008/ and you’ll see your wiki being served up as
freshly rendered HTML! You can also use the -p option to markdoc serve to
control which port it listens on.
IMHO, one of Markdoc’s neatest features is its automatic directory listings. If
you visit http://localhost:8008/_list, a listing of the directory will be
automatically generated, with sub-directories, pages and files displayed (pages
and files also have their file size shown next to their links). If a directory
in your wiki lacks an index page, the listing will become the directory index.
Markdoc’s main hub at the moment is the Bitbucket repository. You can get the source code from there, which contains both the actual utility and all of the documentation as (you guessed it) a Markdoc wiki.
You can submit bug reports via Bitbucket, too, or just e-mail me at zacharyvoase@me.com if you’ve found an issue or have a feature request.
Markdoc (of course) eats its own dogfood. If you’d like to see a
fully-functional wiki in action, the official Markdoc documentation is a good
example. You can find the complete source for the wiki in the doc/ directory
in the repo. To see the
documentation in a browser (assuming you’ve cloned the repo locally):
$ cd markdoc/ # the Markdoc source repository
$ cd doc/
$ markdoc build && markdoc serve
Then just visit http://localhost:8008/ again.