Category Archives: web hooks

Twitter and the Case for Web Hooks

My, my, we do love our Twitter. Tiny messages in a list. A global chatroom really, but just the messages you want to see. What an elegant system.

  • But it’s hard to track conversations.
  • But 140 characters isn’t enough for some dude who has trouble making his thoughts a wee bit more concise and is accustomed to using long, redundant, extra words.
  • But it doesn’t automatically link up my stock symbols in my tweets.

But, but, but.

Enter the API

Everyone who’s used Twitter for a significant amount of time has concocted a list of features that Twitter lacks and desperately needs. Fortunately, Twitter has a given us a nice API to let geeks fill in the gaps somewhat. Without Googling around to find the most recent stat, I’ve read that the majority of tweets are placed through the API instead of the Twitter site, through mobile and desktop apps. That’s a good indication.

Besides API based clients–alternate software that does what the Twitter site does, there has been a surge in Twitter powered bots. These bots read and write to Twitter using some kind of logic in between. I’ve seen autonomous chat bots and autoresponders.

Who Are My Twitter Followers?

A pattern I’ve seen a lot lately with Twitterized apps is “all you have to do to join my app is follow @sometwittername”. @sometwittername then uses the API to find his followers, and then sends the user tweets, or, links via direct message with special links just for that user.  It’s a MLM’ers dream. The top essential tactic of these web marketing guys is to BUILD YOUR LIST, and there’s no friendlier way than Twitter, right?

If you want to see an example of this pattern in action, here’s one, which I reluctantly link to for demonstration purposes only.

Yes, apparently we have “Twitter marketing gurus” now.

So now the developer needs to know who his followers are, so he can spam or provide $ValueAddedService. Currently, the Twitter API gives developers two ways of finding out who is following you. The first one requires polling, which means you have to call the API at regular intervals.

Polling sucks. If you request too often, it’s wasteful. If you don’t request often enough, you miss the opportunity to do cool stuff with new followers in a timely fashion.

A better way to detect new followers is to tap into your email stream. When Twitter emails you with a new follower notification, it tucks away some extra headers into your message. This is documented for some reason in the FAQ. The headers look something like this:

  • X-TwitterEmailType – will be ‘is_following’ or ‘direct_message’
  • X-TwitterCreatedAt – ex: Thu Aug 07 15:17:15 -0700 2008
  • X-TwitterSenderScreenName – ex: ‘bob’
  • X-TwitterSenderName – ex: ‘Bob Smith’
  • X-TwitterSenderID – ex: 12345
  • X-TwitterRecipientScreenName – ex: ‘john’
  • X-TwitterRecipientName – ex: ‘John Doe’
  • X-TwitterRecipientID – ex: 67890
  • X-TwitterDirectMessageID ex: 2346346

So all you have to do to maintain an instantly up-to-date list of your followers is to send your email notifications through a script (most any SMTP server can do this) and check for messages with the X-TwitterEmailType set to ‘is_following’, and then grab the SenderID or SenderScreenName and you’re all set.

This is a wannabe web hook in action.

WTF is a Web Hook?

If this is the first time you’re hearing the term “web hook”, pay attention. It’s the next step in this whole read-write web thing.

In short, a web hook, or “http callback” allows you to subscribe to another website’s events with a URL. You tell a website, “Hey, when some event happens, send some information about it to this URL”. Why? It doesn’t matter, it’s up to the developer to decide what to do with that information.

An excellent example of a web hook is Paypal’s Instant Payment Notification. It works like this:

  1. You come to my store, fill up your cart, and hit the checkout button.
  2. My site sends you and your order info to Paypal, and you pay up.
  3. In my Paypal account, I’ve set up my IPN (web hook!) URL.
  4. Paypal POSTs the order information to the hook URL letting me know that you’ve paid
  5. I record the sale, do whatever business logic I need to do so my Oompa Loompas can ship your stuff to you.

There’s a little more to it, but that’s the basics. I can choose to do whatever I want with that data from Paypal.

The key part is step 5. I do whatever logic I need to do with my own code.

Web hooks are APIs knowing to call other API endpoints. That’s it.

Now back to the Twitter. Why should you have to hook into your email stream to get those “is_following” notifications?  In the same way Twitter sends you an email, it could just as easily send an HTTP POST with that same data to a script you control. Would it not be much simpler to specify a callback URL in your Twitter settings?

Your site would become integrated with Twitter itself. That word integrated is key. Instead a basic request-response API message, now our websites are having a conversation.

I Understand Now! What Possibilities! Web Hooks Rock My Casbah!

Now imagine what would be possible if Twitter allowed hooks for all the different events that happen on the platform.

  • Twitter, ping this URL of mine when someone I follow tweets or DM’s or replies to me.
  • Twitter, ping a different URL of mine when someone unfollows me.
  • Twitter, ping yet another URL when someone I follow updates her profile. She’s cute, and I need to stay on top of those thumbnail changes.

Twitter, just ping my scripts when stuff happens.

Now Make Hooks Programmable

Maybe I’m not interested in every single hookable event. How cool would it be if I could get a list of all my subscribed events through an API call?

  • Me: Yo Twitter, what do I care about? What am I subscribed to?
  • Twitter: Sup dawg, I heard you like profile changes and unfollows, so I send twirps to your tweets so you can twit when I twat.

But instead of jive talk you’d get back XML or JSON with event names and your hook URLs.

Take it one step further and suppose you only want your hook called when only certain people perform certain actions. Why not make setting and unsetting the hooks themselves programmable via the API?

Example: Call my URL when @cutetwitterchick updates her profile.

I could easily perform that logic in my own code, but telling Twitter to do it is that much cooler:

set_hook ( hook_name, hook_URL [, screename | ID | email ]  )
unset_hook ( hook_name, hook_URL [, screename | ID | email ]  )

The Hooked Web

Jump to the future when all of your favorite sites implement programmable hooks. The pipedream, holy grail, end result is that you no longer even need Twitter, because it’s become a protocol. Just like blogs happily send pingbacks, you can install a Twitter-speaking, open sourced package on a Slicehost account that is your own personal Twitter. The protocol is extensible, so you can do things like:

  • display replied messages in threads as conversations
  • allow longer than 140 character messages
  • automatically hyperlink those stock symbols

And, unlike email, newsgroups,  and chatrooms, it doesn’t turn into a jacked-up, spammed to hell system, because you’re still only following what you want to follow. It’s a decentralized, pluggable architecture, and it integrates with any site using web hooks. At your service.

Take this platform one step further and visualize a user interface layer. You have a nice, decentralized activity stream–an open FriendFeed platform with all of your stuff. Maybe it’s organized by the source, or grouped by service type or however you want it organized. Messages can carry their own css styling, but only if you allow it. Messages can have interactive elements, but only if you allow them. And, holy crap, messages can come packaged with next actions! Twitter messages allow you to reply, calling a hook specified in the message. Flickr messages allow you to comment directly from this ‘StreamReader’ software. You subscribe to Mechanical Turk hits, and receive a new hit whenever you complete one.

One more step further. Not only do messages have next actions, but I can write plugins to attach next actions to any message, and I can do it conditionally. This would be like being able to write filters for Google Reader. For example, filter out all the messages from my group of PHP programming buddies when a message is about photography. I’m not interested, but those guys LOVE fucking cameras for some reason.

I’ve very conveniently ignored the implementation details of such an animal. Twitter could quickly become a push architecture with all of these hooks in place, and that turns quickly into n-squared network traffic patterns. Luckily we have some heavy brains working on xmpp and pubsub and the like, and maybe the end solution would be half push, half poll.

At this point it’s just really fun to imagine the possibilities of a read-write-read-write-read-write web.