So I switched our Twitter integration for Kebima from using the HTTP interface to using the XMPP interface. We really wanted real time updates, and polling just seems so barbaric. I found some code on how to create a twitter bot and got the conversion made surprisingly fast. It's still ugly because auto-following has to be done through HTTP, but in a few hours I had a pretty simple bot going.
Then we went to the Web 2.0 Expo and tried it out. And it didn't work. Turns out Twitter's Jabber replies were delayed or somesuch... probably fallout of Twitter's other greyout problems this week. But it got me thinking...
If you had to build twitter from scratch, how thin a veneer over XMPP could you do it with? At its heart, Twitter is a message router. There are interfaces with SMS systems, HTTP, and Jabber. But messages come in, messages go out. I've heard they run XMPP under the hood to handle this, but I've also heard they run rails for a good chunk of functionality. I'm a rails fan as well, but from what I can see, perhaps there's too much rails and not enough ejabberd in the mix.
Conceptually it seems like you could set this up as a set of processes that each act as an internal Jabber client for an individual twitter user. The process is responsible for receiving messages
In addition to this you need gateways for SMS and the HTTP API, but it seems like those could be scaled out fairly easily as they're not user specific. The SMS gateway is just going to build an XMPP message and dispatch it to the matching twitter client. The HTTP gateway is doing the same thing.
The beauty of this is that it's naturally sharded. No shared data between users. You have to parse messages for @responses and route them to appropriate destinations, but it seems like that could easily be written as an ejabberd plugin.
Google has proven this scales to a fairly large user base. Is Twitter already beyond that scale, or am I missing something?
Regardless, we'll be pushing ahead with our plans to add direct GTalk integration for our app. It may go down at times as well, but it seems more robust than Twitter's infrastructure at this point.
Then we went to the Web 2.0 Expo and tried it out. And it didn't work. Turns out Twitter's Jabber replies were delayed or somesuch... probably fallout of Twitter's other greyout problems this week. But it got me thinking...
If you had to build twitter from scratch, how thin a veneer over XMPP could you do it with? At its heart, Twitter is a message router. There are interfaces with SMS systems, HTTP, and Jabber. But messages come in, messages go out. I've heard they run XMPP under the hood to handle this, but I've also heard they run rails for a good chunk of functionality. I'm a rails fan as well, but from what I can see, perhaps there's too much rails and not enough ejabberd in the mix.
Conceptually it seems like you could set this up as a set of processes that each act as an internal Jabber client for an individual twitter user. The process is responsible for receiving messages
- pushing them to SMS or the user's Jabber client
- Building the web page that people visit
- Handling API calls as they come in
- Dispatching messages to followers
In addition to this you need gateways for SMS and the HTTP API, but it seems like those could be scaled out fairly easily as they're not user specific. The SMS gateway is just going to build an XMPP message and dispatch it to the matching twitter client. The HTTP gateway is doing the same thing.
The beauty of this is that it's naturally sharded. No shared data between users. You have to parse messages for @responses and route them to appropriate destinations, but it seems like that could easily be written as an ejabberd plugin.
Google has proven this scales to a fairly large user base. Is Twitter already beyond that scale, or am I missing something?
Regardless, we'll be pushing ahead with our plans to add direct GTalk integration for our app. It may go down at times as well, but it seems more robust than Twitter's infrastructure at this point.
