Dave recently pushed some of my API changes to the main TalkingPuffin. There are quite a few updates. The API is more complete now, more resilient, supports optional REST arguments, and has a method to load all pages of various APIs. I thought I’d show a few of the enhancements here.

This first listing shows how to use the new TwitterArgs class.

package org.talkingpuffin.twitter
 
object ShowAPI {
 
  def main(args: Array[String]) = {
    // set up our credentials and session
    val user = "foo"
    val password = "bar"
    val sess = TwitterSession(user,password)
    // same method as before
    var friendsTweets = sess.getFriendsTimeline();
    System.out.println("got " + friendsTweets.size + " tweets from old style method")
    // use per page count
    friendsTweets = sess.getFriendsTimeline(TwitterArgs.maxResults(200))
    System.out.println("got " + friendsTweets.size + " tweets using per page count")
    // use chained twitter args
    friendsTweets = sess.getFriendsTimeline(TwitterArgs.maxResults(200).page(2))
    System.out.println("got " + friendsTweets.size + " tweets using chained twitter args")
  }
}

There are a variety of methods that support passing in a TwitterArgs instance. These can be constructed by calling the various methods on the TwitterArgs object, e.g.

val args = TwitterArgs.maxResults(200)

If you want to pass in multiple optional arguments you can make calls on an existing args instance, e.g.

val args = oldArgs.page(2)

These get converted to a URL query segment, and are appended to the URLs called for twitter data.

The next listing shows some Scala functional programming neatness, and builds on my previous post about retry logic.

package org.talkingpuffin.twitter
 
object ShowAPI {
 
  def main(args: Array[String]) = {
    // set up our credentials and session
    val user = ""
    val password = ""
    val sess = TwitterSession(user,password)
 
    // demo load all... this just loads the first page
    var myTweets = sess.getUserTimeline("mccv")
    System.out.println("got " + myTweets.size + " tweets from my timeline")
    // now we show load all.  loadAll just wants a function that takes an int as an arg,
    // which is the page.  Scala's partially applied functions make this pretty easy
    // to use in a general purpose way
    myTweets = sess.loadAll(sess.getUserTimeline("mccv",_:Int))
    System.out.println("got " + myTweets.size + " tweets using loadAll")
    // this is even fancier.  Here I add retry logic to load all.
    // note that retryPage is a function I defined here... but the session
    // doesn't care.  It keeps iterating through pages, retrying and loading
    // until it reaches the end.
    myTweets = sess.loadAll(retryPage(_:Int,sess.getUserTimeline("mccv",_:Int)))
    System.out.println("got " + myTweets.size + " tweets using loadAll and retries")
  }
 
    /**
    * this is a function that is sort of a thunk through to tryNTimes.
    */
    def retryPage[T](page:Int, func: (Int) => T):T = {
        // here we define a privately scoped function
        // that can be passed to tryNTimes
        def tryPage() = {
            func(page)
        }
        // and now we try N (5) times
        tryNTimes(tryPage,5)
    }
 
    /**
    * from the last blog post, a retrier
    */
    def tryNTimes[T](func: () => T, runNumber: Int):T = {
      try{
          func()
      } catch {
        case e if runNumber > 1 => tryNTimes(func,runNumber - 1)
        case e => throw e
      }
    }
 
}

Hopefully this code is more or less self documenting. The first session call just gets the first page of the user timeline. This is usually sufficient for writing a Twitter client, but if you are doing data mining it isn’t so great. The new API introduces a method called loadAll of type (f:(Int) => List[T]) => List[T]. This means that any method that takes a single int argument (a page number) and returns a list can be passed to loadAll. It keeps executing the passed in function with increasing page numbers until an empty list is returned (note that this must be the behavior on page overruns as currently implemented. If the overrun URI returns a 404 we’ll get an exception thrown. Luckily Twitter currently just returns an empty list).

The second call shows this in action. It’s using a slightly more complicated case, because getUserTimeline takes a String and an Int. Scala’s partially applied functions make this a snap. The line

sess.getUserTimeline("mccv",_:Int)

Takes the getUserTimeline call with one bound argument and one unbound. It returns a function of type (Int) => List[TwitterStatus], which is exactly what loadAll wants.

The third call is even more complicated um, sophisticated. Let’s say we want to retry operations five times, just in case we get dropped connections in the middle of a big load. Well, all we need to do is get a function that takes an int and returns a list into loadAll.

In a previous blog post I wrote about implementing a retryable method. You can see this more or less unchanged at the end of the file. Unfortunately its signature isn’t quite what we want. So we define retryPage, which acts as an adapter from loadAll to tryNTimes. With this setup in place, we can set up our last call, which uses two partially applied functions. The first converts getUserTimeline into the page-argument-only form, and the second converts retryPage into a page-argument-only form.

Running the two samples combined this gives us the following output

got 20 tweets from old style method
got 199 tweets using per page count
got 200 tweets using chained twitter args
got 20 tweets from my timeline
got 824 tweets using loadAll
trying to get page 1
trying to get page 2
trying to get page 3
trying to get page 4
trying to get page 5
trying to get page 6
trying to get page 7
trying to get page 8
trying to get page 9
trying to get page 10
trying to get page 11
trying to get page 12
trying to get page 13
trying to get page 14
trying to get page 15
trying to get page 16
trying to get page 17
trying to get page 18
trying to get page 19
trying to get page 20
trying to get page 21
trying to get page 22
trying to get page 23
trying to get page 24
trying to get page 25
trying to get page 26
trying to get page 27
trying to get page 28
trying to get page 29
trying to get page 30
trying to get page 31
trying to get page 32
trying to get page 33
trying to get page 34
trying to get page 35
trying to get page 36
trying to get page 37
trying to get page 38
trying to get page 39
trying to get page 40
trying to get page 41
trying to get page 42
trying to get page 43
got 824 tweets using loadAll and retries

Last week I ran into a cose where needed to add some logic to retry certain operations a number of times before failing. These operations can throw a variety of exceptions, some of which are recoverable, some of which are not. Due to a variety of issues the code in question must be Java. At least for the time being. So I set out to do this. The solution I came up with wasn’t bad as Java solutions go…

Step 1: identify recoverable exceptions. Because I control the code that can throw the exceptions this isn’t so bad. I create a simple RecoverableException class.

package org.mccv;
 
/**
 * A marker class for recoverable exceptions
 */
public class RecoverableException extends Exception{
 
}

Step 2: create an interface that defines a retryable operation.

package org.mccv;
 
/**
 * An interface that defines an execute operation to retry
 */
public interface Retryable<T> {
	public T execute() throws Exception;
}

Step 3: implement a class that will retry a retryable n times before failing.

package org.mccv;
 
/**
 * The implementation of our retrying class
 */
public class Retryer<T> {
	/**
	 * The retryable we will call
	 */
	private Retryable<T> retryable;
 
	/**
	 * Just assign our retryable field 
	 */
	public Retryer(Retryable<T> retryable){
		this.retryable = retryable;
	}
 
	/**
	 * Try to execute our retryable n times 
	 */
	public T tryTimes(int times) throws Exception{
		// store the last thrown recoverable exception
		RecoverableException lastException = null;
		// try the specified number of times
		for(int i = 0; i < times; i++){
			System.out.println("running it with " + (times-i) + " tries remaining");
			try{
				return retryable.execute();
			}catch(RecoverableException re){
				lastException = re;
			}
		}
		throw lastException;
	}
}

Step 4: implement the retryable interface with the operation we want retried

package org.mccv;
 
/**
 * An implementation of the Retryable interface
 */
public class RetryableImpl implements Retryable<String>{
	/**
	 * Show usage of the retryer
	 */
	public static void main(String [] args){
		// create a new retrier
		Retryer<String> retrier = new Retryer<String>(new RetryableImpl());
		// run the retrier 
		try{
			System.out.println("result = " + retrier.tryTimes(3));
		}catch(Exception e){
			System.out.println("failed after 3 tries");
		}
	}
 
	/**
	 * An intentionally flaky operation
	 */
	public String execute() throws Exception{
	    if(Math.random() > 0.3){
	        throw new RecoverableException();
	      }else{
	        return "foo";
	      }
 
	}
}

This works, and you can use anonymous inner classes to wrap stuff that doesn’t directly extend retryable. But it’s ugly. You have to make a lot of things final you wouldn’t otherwise (for instance let’s say you want your retryable to set a status code and response message. This gets difficult due to having to mark things final). It’s also a bit clumsy in that you have a Retryer class in play, plus an implementation of Retryable. What I really wanted was a way to say

retry(myOperation,nTimes).

With Scala I got a lot closer.

package org.mccv
 
/**
 * A marker trait indicating an exception that is recoverable.
 * Note that we could make this a trait as well.
 */
class RecoverableException extends Exception
 
/**
 * Usage of our retryable class
 */
object Retryer {
	/**
	* The workhorse.  Runs the operation, and if successful returns the result.
	* If unsuccessful calls itself recursively with a decremented run number.
	*/
	def tryNTimes[T](func: () => T, runNumber: Int):T = {
	  println("running it with " + runNumber + " tries remaining")
	  try{
		  func()
	  } catch {
	    case e:RecoverableException if runNumber > 1 => tryNTimes(func,runNumber - 1)
	    case e => throw e
	  }
	}
}
 
object Main {
   def main(args: Array[String]):Unit = {
    val tries = 3;
    try{
    	println(Retryer.tryNTimes[String](flakyMethod,tries))
     }catch{
       case e => println("failed after " + tries + " tries")
     }
  }
 
  def flakyMethod():String = {
    if(Math.random > 0.3){
      throw new RecoverableException()
    }else{
      "foo"
    }
  }
}

This is almost exactly what I’d like. I define a simple object that has a tryNTimes method. Because it’s an object it’s sort of like a Java static method… I can call Retryer.tryNTimes() directly. This is a recursive call. If a RecoverableException is caught, it just calls tryNTimes with the number of tries decremented. The beauty of this is that you can pass in any zero argument function. It doesn’t matter what class it’s defined in… it just has to know about recoverable exceptions.

I’ll be keeping an eye out for things that are easier in scala and keeping this series up to date. I’m sure I’ll be running into more of them.