Twitter and the Case for Web Hooks

My, my, we do love our Twitter. Tiny messages in a list. A global chatroom really, but just the messages you want to see. What an elegant system.

  • But it’s hard to track conversations.
  • But 140 characters isn’t enough for some dude who has trouble making his thoughts a wee bit more concise and is accustomed to using long, redundant, extra words.
  • But it doesn’t automatically link up my stock symbols in my tweets.

But, but, but.

Enter the API

Everyone who’s used Twitter for a significant amount of time has concocted a list of features that Twitter lacks and desperately needs. Fortunately, Twitter has a given us a nice API to let geeks fill in the gaps somewhat. Without Googling around to find the most recent stat, I’ve read that the majority of tweets are placed through the API instead of the Twitter site, through mobile and desktop apps. That’s a good indication.

Besides API based clients–alternate software that does what the Twitter site does, there has been a surge in Twitter powered bots. These bots read and write to Twitter using some kind of logic in between. I’ve seen autonomous chat bots and autoresponders.

Who Are My Twitter Followers?

A pattern I’ve seen a lot lately with Twitterized apps is “all you have to do to join my app is follow @sometwittername”. @sometwittername then uses the API to find his followers, and then sends the user tweets, or, links via direct message with special links just for that user.  It’s a MLM’ers dream. The top essential tactic of these web marketing guys is to BUILD YOUR LIST, and there’s no friendlier way than Twitter, right?

If you want to see an example of this pattern in action, here’s one, which I reluctantly link to for demonstration purposes only.

Yes, apparently we have “Twitter marketing gurus” now.

So now the developer needs to know who his followers are, so he can spam or provide $ValueAddedService. Currently, the Twitter API gives developers two ways of finding out who is following you. The first one requires polling, which means you have to call the API at regular intervals.

Polling sucks. If you request too often, it’s wasteful. If you don’t request often enough, you miss the opportunity to do cool stuff with new followers in a timely fashion.

A better way to detect new followers is to tap into your email stream. When Twitter emails you with a new follower notification, it tucks away some extra headers into your message. This is documented for some reason in the FAQ. The headers look something like this:

  • X-TwitterEmailType - will be ‘is_following’ or ‘direct_message’
  • X-TwitterCreatedAt - ex: Thu Aug 07 15:17:15 -0700 2008
  • X-TwitterSenderScreenName - ex: ‘bob’
  • X-TwitterSenderName - ex: ‘Bob Smith’
  • X-TwitterSenderID - ex: 12345
  • X-TwitterRecipientScreenName - ex: ‘john’
  • X-TwitterRecipientName - ex: ‘John Doe’
  • X-TwitterRecipientID - ex: 67890
  • X-TwitterDirectMessageID - ex: 2346346

So all you have to do to maintain an instantly up-to-date list of your followers is to send your email notifications through a script (most any SMTP server can do this) and check for messages with the X-TwitterEmailType set to ‘is_following’, and then grab the SenderID or SenderScreenName and you’re all set.

This is a wannabe web hook in action.

WTF is a Web Hook?

If this is the first time you’re hearing the term “web hook”, pay attention. It’s the next step in this whole read-write web thing.

In short, a web hook, or “http callback” allows you to subscribe to another website’s events with a URL. You tell a website, “Hey, when some event happens, send some information about it to this URL”. Why? It doesn’t matter, it’s up to the developer to decide what to do with that information.

An excellent example of a web hook is Paypal’s Instant Payment Notification. It works like this:

  1. You come to my store, fill up your cart, and hit the checkout button.
  2. My site sends you and your order info to Paypal, and you pay up.
  3. In my Paypal account, I’ve set up my IPN (web hook!) URL.
  4. Paypal POSTs the order information to the hook URL letting me know that you’ve paid
  5. I record the sale, do whatever business logic I need to do so my Oompa Loompas can ship your stuff to you.

There’s a little more to it, but that’s the basics. I can choose to do whatever I want with that data from Paypal.

The key part is step 5. I do whatever logic I need to do with my own code.

Web hooks are APIs knowing to call other API endpoints. That’s it.

Now back to the Twitter. Why should you have to hook into your email stream to get those “is_following” notifications?  In the same way Twitter sends you an email, it could just as easily send an HTTP POST with that same data to a script you control. Would it not be much simpler to specify a callback URL in your Twitter settings?

Your site would become integrated with Twitter itself. That word integrated is key. Instead a basic request-response API message, now our websites are having a conversation.

I Understand Now! What Possibilities! Web Hooks Rock My Casbah!

Now imagine what would be possible if Twitter allowed hooks for all the different events that happen on the platform.

  • Twitter, ping this URL of mine when someone I follow tweets or DM’s or replies to me.
  • Twitter, ping a different URL of mine when someone unfollows me.
  • Twitter, ping yet another URL when someone I follow updates her profile. She’s cute, and I need to stay on top of those thumbnail changes.

Twitter, just ping my scripts when stuff happens.

Now Make Hooks Programmable

Maybe I’m not interested in every single hookable event. How cool would it be if I could get a list of all my subscribed events through an API call?

  • Me: Yo Twitter, what do I care about? What am I subscribed to?
  • Twitter: Sup dawg, I heard you like profile changes and unfollows, so I send twirps to your tweets so you can twit when I twat.

But instead of jive talk you’d get back XML or JSON with event names and your hook URLs.

Take it one step further and suppose you only want your hook called when only certain people perform certain actions. Why not make setting and unsetting the hooks themselves programmable via the API?

Example: Call my URL http://myawesometwittersite.com/myhooks/profile/update when @cutetwitterchick updates her profile.

I could easily perform that logic in my own code, but telling Twitter to do it is that much cooler:

set_hook ( hook_name, hook_URL [, screename | ID | email ]  )
unset_hook ( hook_name, hook_URL [, screename | ID | email ]  )
list_hooks()

The Hooked Web

Jump to the future when all of your favorite sites implement programmable hooks. The pipedream, holy grail, end result is that you no longer even need Twitter, because it’s become a protocol. Just like blogs happily send pingbacks, you can install a Twitter-speaking, open sourced package on a Slicehost account that is your own personal Twitter. The protocol is extensible, so you can do things like:

  • display replied messages in threads as conversations
  • allow longer than 140 character messages
  • automatically hyperlink those stock symbols

And, unlike email, newsgroups,  and chatrooms, it doesn’t turn into a jacked-up, spammed to hell system, because you’re still only following what you want to follow. It’s a decentralized, pluggable architecture, and it integrates with any site using web hooks. At your service.

Take this platform one step further and visualize a user interface layer. You have a nice, decentralized activity stream–an open FriendFeed platform with all of your stuff. Maybe it’s organized by the source, or grouped by service type or however you want it organized. Messages can carry their own css styling, but only if you allow it. Messages can have interactive elements, but only if you allow them. And, holy crap, messages can come packaged with next actions! Twitter messages allow you to reply, calling a hook specified in the message. Flickr messages allow you to comment directly from this ‘StreamReader’ software. You subscribe to Mechanical Turk hits, and receive a new hit whenever you complete one.

One more step further. Not only do messages have next actions, but I can write plugins to attach next actions to any message, and I can do it conditionally. This would be like being able to write filters for Google Reader. For example, filter out all the messages from my group of PHP programming buddies when a message is about photography. I’m not interested, but those guys LOVE fucking cameras for some reason.

I’ve very conveniently ignored the implementation details of such an animal. Twitter could quickly become a push architecture with all of these hooks in place, and that turns quickly into n-squared network traffic patterns. Luckily we have some heavy brains working on xmpp and pubsub and the like, and maybe the end solution would be half push, half poll.

At this point it’s just really fun to imagine the possibilities of a read-write-read-write-read-write web.

About these ads

About John Herren

John Herren is a developer and technical consultant with focus on web applications. He currently serves as Director of Development for Primetime US, the company behind the hit movie and book The Secet. John was formerly staff writer and developer community evangelist for Zend Technologies. Along with founding neat experiments like TagCloud.com, John is an active member in the mashup community, working with API providers and speaking at conferences. He is a published author of Linux certification study material. John enjoys using open source software like PHP and Ruby on Rails to bend the web into exciting new chimeras of hyperlinked goodness. View all posts by John Herren

18 responses to “Twitter and the Case for Web Hooks

  • David Berlind

    Masterful John. The most salient point of all this is at the end. I’ve noticed how almost everyone I pay attention to on Twitter is interacting with a 3rd party app and thinking, if the Twitter guys aren’t careful, they could get disintermediated. But, even though I’m a protocol guy, it never dawned on me that the Twitter protocol could be just like SMTP is for email withour insourced Twitter servers talking to each other, or, given where things are going today, our outsourced twitter servers in the cloud (be it slicehost, ec2 or whatever).

    Brilliant. My mind is cracked open. Where are the open source developers on this one?

  • Terry Jones

    Hi John

    Thanks for writing all this up, it’s thought-provoking. I’ll be thinking about it for some time, I know.

    One issue that comes immediately to mind is authentication. When you generate a URL for a pingback, it may need to include auth information and may also need to be unguessable via a random component. For some uses I think that’s fine – guessing 128 bit random numbers is hard. But in other cases it’s not. So a use case like a pingback from Paypal to a time-limited randomized URL seems fine, whereas a URL that can be used repeatedly to tell you of an event on (say) Twitter is less good. The multi-use URL might be guessable, it might leak out from Twitter, etc.

    You might be interested in FluidDB, a REST-addressable always writable database we’re building. It would be easy to be handing out URLs to do things to objects in FluidDB (but see above comments re auth, etc) instead of running your own architecture, keeping it up. FluidDB is intended to be a very flexible information store (providing search too, of course), and one that is always writable by anyone. I thought years ago about implementing some form of hook functionality (a la emacs), but have not tried it because we’ve got our work cut out just building the simplest possible underlying architecture. But we may get there.

    Anyway, as I said and as you know, there’s tons to think about here. I’m 100% convinced that there are major changes ahead in how we work with information, based on making the underlying architecture more writable.

    OK, enough for now. Thanks again.

    @terrycojones

  • Jeff Lindsay

    When I first started talking to Twitter about web hooks in 2007 (actually, they were the first startup that came to me saying they’d use them after a talk I gave), they were really excited and envisioned themselves being more like a generalized message exchange. I think things have changed since then.

  • Loath To Improve Itself, Could Twitter Be Disintermediated By A Protocol?

    […] spotted opportunity at the intersection of Twitter and what he calls “Web hooks.” In a blog he posted earlier today, he offers some real-world examples of how easy it is to trap […]

  • links for 2009-03-05 « Bloggitation

    […] Twitter and the Case for Web Hooks (tags: programming web2.0 twitter webhooks) […]

  • Jason Garber

    John, this is an excellent blog piece. I wish there was more of the “big picture” being presented out there.

    The concept of hooks that you articulated so well has been used internally at my company for some time — let the programmer write a simple callback and you open a world of possibilities.

    Now, expand it to the web… Watch out…

  • Neil Garb

    I recently launched a very simple web service that polls pull services, such as RSS, and pushes updates to an endpoint of your choice, essentially simulating a web hook where none exists.

    I just completed a basic Twitter adapter.

    http://myqron.com

  • John Herren

    @Neil, I saw your announcement via the web hooks mailing list. Thanks for posting your link here.

  • John Kalucki

    Check out the Twitter Streaming API documentation at: http://apiwiki.twitter.com/Streaming-API-Documentation. There are, for example, resources for following all updates from an arbitrary list of users.

    Also, you can use the Search API to perform some similar push-like operations.

  • Twitter Hooks « blogrium

    […] URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be […]

  • guruofsales

    That is just a masterpiece tutorial in my opinion :-)

    Bests,
    Alex (GuruOfSales)

  • nunobaba

    Two scenarios I thought about. 1/ In a wild dream, Google Docs pings me whenever a doc I shared with my team is updated. That event could be include in my sort of activity stream. 2/ The read-write-read-write-read-write web scheme would end up in a near real-time conversation, punctuated by various sorts of events I’ve set up.

    My question is about the last action. I still don’t get is how you would refresh your stream of activities dynamically, without constantly polling your urls?

    Thanks for your reply and great write-up.

  • Sousa Martin

    Any update on webhooks for twitter? You make a strong case.

  • Jeff Lindsay

    Sousa: http://www.twitterhooks.com

    Still in “alpha” … but keep an eye on it.

  • What is the “pushbutton web”? - TechnoPhobia

    […] been a fair bit of talk about webhooks and the ‘real-time web’ this year and a recent post from Anil Dash caught my attention as it […]

  • » Blog Archive » APIs, TOS, and building a hooked web

    […] on Webhooks last year) to speak at Gluecon. Simultaneously, I’ve been reading blog posts like this — where the punch-line is: Jump to the future when all of your favorite sites implement […]

  • APIs, TOS, and building a hooked web | CloudAve

    […] on Webhooks last year) to speak at Gluecon. Simultaneously, I’ve been reading blog posts like this — where the punch-line is: “Jump to the future when all of your favorite sites implement […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: