Probably Programming

I’ve been neglecting both walking and posting on here, though I’ve neglected posting more, obviously.

I’ve been very busy with work and side projects and my house trying to fall apart on me, so I’ve been out of free time lately.

Anyway, my total is now up to 45 miles of walking. Not too much more than last time, really. I’ve gone on some walks around the neighborhood with Angie, and I’m not counting them, mostly because I don’t know how long they are, really.

I decided I wanted to start a “chain,” where every day I do something to work toward a goal I’ve set for myself (yes, I mean other than the walking, though that can be a chain as well).

Fortunately enough for me, I found Don’t Break The Chain, where you can keep track of one or more “chains” of things you want to do every day.

The Twitter Client Chain

So I’m creating a web-based Twitter client, as a project for me to try out all the different techniques and technologies that I want to use, but don’t have the chance to work with as much as I’d like during my day job. Also, ideally, it will at some point become a supplementary source of income.

That’s a quite a way off until I have more free time in a month or two, but for now, I will satisfy myself with making some small amount of progress on it each day (hopefully).

I’m writing this client in Erlang, using the awesome Nitrogen web framework with the equally awesome MochiWeb web server. My goals for this project include using test driven development and continuous deployment and fully embracing the mantra of “release early, release often”.

The plan is to have something up and running on a server as soon as possible. By something, I mean even less than a full-featured twitter client. But as soon as something is up, I can start getting feedback, and see what I should work on next. If I’m creating a product for users, there’s no use in developing what I want.

I need to develop what my users want.

I started this chain up about a week ago, and broke it after the first day. Now I’m trying again.

Tests and pre-commit hooks

During my previous, one-day chain, I managed to get a test harness (using the EUnit Erlang unit test framework) set up which will find any modules ending with _tests, stopping at the first error.

At the moment, I have a shell script which runs the tests. When it starts to take too long, I will switch over to an Erlang module which loops through, so I don’t get the startup times of the Erlang interpreter for each test. Or if anyone knows an easier way (perhaps there’s already a module that does that somewhere?).

#!/bin/bash
 
for modfile in `ls ebin/*_tests.beam`
do
    mod=`basename $modfile .beam`
    echo Running $mod:
    erl -noshell -pa ./ebin -s runtest main ${mod%.*}
    [ $? -eq 0 ] || exit 1
done
[ $mod ] || exit 1

The runtest module consists of the following:

-module(runtest).
-export([main/1]).
 
main(Mod) ->
    case eunit:test(Mod) of
        error -> halt(1);
        _ -> init:stop()
    end.

It’s all kind of thrown together, but the point is incremental improvement, so I’ll get back to it when/if I need to. Like I said, there’s probably something else out there that will do a better job at this than I did.

So, today, my “chain link” was partially taking that script and setting it up to run from the GIT pre-commit script, and partially writing this blog post.

Getting it set up to run as a pre-commit script just involved making sure it returns non-zero whenever there is a test error (or if no tests were run, which happens when make hasn’t been run yet).

The pre-commit script itself (placed in .git/hooks/pre-commit and set executable) is pretty simple:

#!/bin/sh
./tests.sh
if [ $? != 0 ]
then
        echo "===================================="
        echo " Commit stopped due to test failure"
        echo "===================================="
        exit 1
fi

Chain length: 1!

What’s next?

Next up, I’m acutally going to start on the meat of the project. Since I haven’t fully decided on what GUI library I want to use, nor what GUI testing framework I want to use, I’m going to start on the backend with code to manage user login info and queries (timeline searches, DMs, etc.).

I’ve got some great ideas (I think) on ways to enhance the way people have conversations over Twitter (and eventually other mediums), so you can be sure that there will be more posts about this in the future.

This is a really simple tip, but it was handy for me, and it could be helpful for someone else, too.

While playing around with Nitrogen, which is a great project by the way (more in a later blog post, I guarantee it), I noticed that I had to restart the server to re-load modules as I changed them (or I could manually, reload them, I know, I know..).

Since I was using Mochiweb as my backend, and I’m used to the development version of Mochiweb auto-reloading modules as you recompile, I wanted the same functionality here.

Turns out it’s as simple as changing the default start.sh from

#!/bin/sh
cd `dirname $0`
 
echo Starting Nitrogen.
erl \
	-name nitrogen@localhost \
	-pa ./ebin -pa ./include \
	-s make all \
	-eval "application:start(appname)"

to this

#!/bin/sh
cd `dirname $0`
 
echo Starting Nitrogen.
erl \
	-name nitrogen@localhost \
	-pa ./ebin -pa ./include \
	-s make all \
	-s reloader \
	-eval "application:start(appname)"

… and bam, modules now reload as you recompile them.

There has been some talk about and arguments against and responses to issues about using rev=”cononical” for referencing shorter URLs instead of the automated use of TinyURL when posting to sites like Twitter.

I must say that I agree with Ben Ramsey (see “arguments agains” above) in suggesting we use rel=”alternate shorter” instead.

I also like the idea that Chris Shiflett had of using a HTTP header and a HEAD request to make it so you neither have to retrieve the entire requested page nor parse any HTML. I’d stick with Ben’s suggestion, however, and make the header something like “X-Alternate-Shorter:”, rather than “X-Rev-Canonical”. What’s the harm in calling it something that actually makes sense?

The idea of using HTTP HEAD requests to solve the problem inspired me to come up with a more immediate solution to one of the problems introduced by using url shortening services: uncertainty about where a URL leads.

This problem can be solved on the client side, which requires no work on the part of Twitter (meaning this is more likely to be put into use sooner).

Since most URL shortening services use an HTTP redirect to do their job, all it takes is a HEAD request to the tiny URL in question, and then a look at whatever “Location:” header is returned to see what the real URL is. In fact, you don’t even really need to do a HEAD request in most cases, since most URL shortening services don’t return any body, since they are just redirecting you anyway.

Read on for more information and implementations of an untinyurl function in various languages.

(more…)

I’m working on a little project in Erlang and I wanted to use only OpenID for my authentication. It turns out there is currently no Erlang OpenID consumer library (or if there is, I couldn’t find it).

So I’ve started writing my own. So far I’ve got the first necessary step complete: HTML-based discovery.

I’m starting with version 1.1, simply because it is shorter and requires less (I don’t want to implement the XRI or Yadis protocols just yet).

It turns out that mochiweb comes with an HTML parser, so I used that, since I’m using mochiweb for my application. The parsed HTML comes back as a series of nested tuples of the format {<<”tag”>>, Attributes, Children}, where “tag” is the tagname (the root will be <<”html”>>, for example), Attributes is a proplist of that tag’s attributes, and Children is a list of more tuples of the same format and/or binaries with the contents of text nodes. Everything is represented as binaries, so I use those directly rather than converting between strings and binaries.

Here’s the code which finds the link tags with rel=”openid.server” and rel=”openid.delegate” (if it is there):

get_openid_server(Identifier) ->
    NormalizedIdentifier = normalize_identifier(Identifier),
    case http:request(NormalizedIdentifier) of
        {ok, {_Status, _Headers, Body}} ->
            HtmlTokens = mochiweb_html:parse(Body),
            find_openid_tags(HtmlTokens);
        _ ->
            {error, http_error}
    end.
 
normalize_identifier(Ident = "http://" ++ _Rest) ->
    Ident;
normalize_identifier(Ident) ->
    "http://" ++ Ident.
 
find_openid_tags(HtmlTokens) ->
    case find_tag(<<"head">>, [HtmlTokens]) of
        {<<"head">>, _Attrs, Children, _Rest} ->
            case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.server">>}, Children) of
                not_found ->
                    {error, openid_server_not_found};
                ServerAttrs ->
                    Server = proplists:get_value(<<"href">>, ServerAttrs),
                    case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.delegate">>}, Children) of
                        not_found ->
                            [{server, Server}];
                        DelegateAttrs ->
                            Delegate = proplists:get_value(<<"href">>, DelegateAttrs),
                            [{server, Server}, {delegate, Delegate}]
                    end
            end;
        not_found ->
            {error, no_head_tag}
    end.
 
find_tag(_TagName, []) ->
    not_found;
find_tag(TagName, [{TagName, Attributes, Children} | Rest]) ->
    {TagName, Attributes, Children, Rest};
find_tag(TagName, [{_OtherTag, _Attributes, Children} | Rest]) ->
    find_tag(TagName, Children ++ Rest);
find_tag(TagName, [_Other | Rest]) ->
    find_tag(TagName, Rest).
 
find_tag_with_attr(_TagName, {_AttrKey, _AttrVal}, []) ->
    not_found;
find_tag_with_attr(TagName, Attr = {AttrKey, AttrVal}, Tags) ->
    case find_tag(TagName, Tags) of
        not_found ->
            not_found;
        {TagName, Attributes, Children, Rest} ->
            case proplists:get_value(AttrKey, Attributes) of
                AttrVal ->
                    Attributes;
                _ -> 
                    find_tag_with_attr(TagName, Attr, Children ++ Rest) 
            end
    end.

(the <PIPE>s above should be “|”s and the <SEMI>s should be “;”s. Not sure why the syntax highlighter is doing that to them…

Assuming the above code is put into a module called “openid”, you get the following:

1> openid:get_openid_server("blog.paulbonser.com")
[{server,<<"http://www.livejournal.com/openid/server.bml">>},
 {delegate,<<"http://misterpib.livejournal.com/">>}]
2>

As I said, this is the first step. Hopefully I’ll have some time very soon to get on with the next couple of steps, and then I’ll be done.

Much to my surprise, my last post generated more traffic in a single day than my blog has ever gotten in a single month. Apparently people are quite interested in making web applications with Python. I’ve started on part two, but since so many people showed interest I want to spend more time on it than I spent on the last one. So instead, you get this post.

So I’ve been fiddling around with CouchDB lately. Since it’s common to store tree-based data, and it’s kind of a pain to do so in your standard relational DB, I thought it would be a good exercise to see how hard it is to store hierarchical data in CouchDB.

Turns out it’s pretty easy.

(more…)