Using data from web APIs in Isadora: part 1

Posted by on 13 March 2014 | Comments

What is a web API?
Web API's are sets of communication channels that are defined by a web application/service for the input and output of data. API strictly means: application programming interface.
Using web API's we can interact with data from a web service. View the latest Twitter post, or check how many people like your Facebook page. Really the possibilities are huge, and I am not here to try to suggest cool ways to use web data with Isadora, but rather to introduce the Isadora actors required to communicate with an API.

p1 apisNow, not all API's are the same, infact; most differ greatly, and what works for one may not work for another. With this in mind I am starting simple. We will use the test API available at jsontest.com, is very simple to get started with, doesn't require authentication (see Oauth required for Facebook and Twitter) and supplies enough data that we can quickly learn how to parse the data returned by the TCP-IP request.

In this tutorial 'Part 1' we are going to make a tcp-ip GET request. You may want to read a little about HTTP request methods, the short of it is that we need to GET information, and we need to POST information to the system.
Starting with GET is the easiest.

To get started  I assume you have Isadora installed and know how to use it.
We will need to bring a few actors into the patch windows.

  1. TCP Send Data
  2. TCP In Watcher - Text (there is also a Binary version but we are not using it here)
  3. TCP Stream Control
p1 a

in place and connected

Once you have these 3 actors on stage you can go ahead and connect the 'msg rcv' outlet of the 'TCP In Watcher - Text' actor to the 'deactivate' inlet of the 'TCP Stream Control' actor.
That is all the patch connecting we will do to have a GET request work.

Now lets start with the 'TCP Send Data' actor, we set the inputs as follows:

  • Stream id = 1
  • Params = 1 (we are not using these this time)
  • Param 1 = 0 (default)

Next we need to define what TCP request is being made. Double click the 'TCP Send Data' actor, a small text window should open up. The default text in the window is "Data", replace this text with:  

 "GET /?callback=showMyIP HTTP/1.1" 0D 0A "HOST: ip.jsontest.com" 0D 0A 0D 0A

What is that? OK, this is a HTTP request, I am not going to go into much detail here, except to say that this exerpt from that site "HTTP 1.1 defines 46 headers, and one (Host:) is required in requests." will help us understand this GET request a little better.

p1 b

GET request entered

The first part of the request: "GET /?callback=showMyIP HTTP/1.1"
defines the Method, the path, and the HTTP version.
The method = GET
The path = /?callback=showMyIP
The HTTP version = HTTP/1.1

This segment is pretty straight forward in that there are 3 clear blocks of data that are separated by a single space.

Next we see the cryptic looking "0D 0A", this is equivalent to "\r\n", or Carriage return, Line feed. Its telling the system that the initial line is complete.

So on to "HOST: ip.jsontest.com", as you might guess this defines the host that we will be sending the GET request to, and it is required because we are using HTTP/1.1
We are using the domain name. but could just as easily use the servers IP address.
I guess in theory we could change the HTTP version, and drop the host parameter, but I haven't tried it, and I don't see any reason to use older protocols.

OK, its maybe worth noting at this point that:
"GET /?callback=showMyIP HTTP/1.1" 0D 0A "HOST: ip.jsontest.com" 0D 0A 0D 0A
works the same as:
"GET /?callback=showMyIP HTTP/1.1\r\nHost: ip.jsontest.com\r\n\r\n"
in the 'TCP Send Data' actor.

 Looking at the differences between these 2 text strings can help you understand how the 'TCP Send Data' actor is reading this, and this will become important in a later section.

Now you might be asking why is there a \r\n for each line? Wouldn't just a single \n for each new line work? Yes, in my tests you can in-fact use just a single \n instead of the \r\n, however; the HTTP specification includes a carriage return and new line, both the \r and \n or as 0D 0A. So in an attempt to avoid any possible future errors or server incompatibilities I am sticking to the \r\n format.

We now have a correctly formatted GET request! That's great.
Its a good time to review the communication sequence in more detail. The steps in the sequence go like this:

  1. Open connection
  2. Wait for server to respond
  3. Connection is confirmed by server
  4. Send GET request to server
  5. Wait for server to respond
  6. Data is received
  7. Server closes connection

By triggering the 'TCP Send Data' actor we initiate Step 4, but how do we do steps 1 thru 3?

We need to setup the 'TCP Stream Control' actor. This task is much easier than the 'TCP Send Data' actor, so we will get thru this quickly.

p1 c

TCP Stream Control : Setup

Set the actors inputs as follows:

  • Stream id = 1 (and integer should work)
  • Tcp addr = Set this to match the Host from the GET request (we are using: ip.jsontest.com)
  • Tcp post = 80 (standard port for http communications)
  • Buf size = Default is good (I set mine to 350, as testing shows that this is the minimum that will work, you can use the Monitor window to see the full server reply in both text and Hex (Windows/Show Monitor), remember we are watching the tcp-ip data. You will then be able to see why 350 bytes are required.)

Good, that's it for that actor. We now need to setup the final actor 'TCP In Watcher - Text'.
Again, since we connected the 'msg rcv' output to the 'deactivate' input of the 'TCP Stream Control' actor at the beginning of this tutorial all we need to do is set the 3 inputs:

  • Stream id = 1 (or an integer to match the 'TCP Stream Control' actors id)
  • Enable = On
  • Eom char = 10 (line feed, More ascii codes: note the Hex codes for line feed and carriage return)
p1 d

Text parsing code

And setup the TCP Input Parser. If you double click the 'TCP In Watcher - Text' another text edit window will pop open. This one defaults to blank, and I recommend you click the help button and read thru the docs. Isadora handles text parsing in a number of actors using the same formatting, it reminds me of regular expressions and is rather powerful and easy to use once you understand it.
Enter the following in the text window:

msg : string = [^;] ";"

This little piece of code is doing only a few things. It is first defining the output of the actor. If you close the text window, you should see a 'msg' output as part of the 'TCP In Watcher - Text' actor now. Next is defines the output as a string, and sets the string equal to the parsed data.

The parsed data? I know that didn't explain anything, so here it is.
We start the parsing with a selector, [^;] which says select any (^) char except ';', then we have the output def that says output this until ';'
Basically if you set the char following the '^' and inside the "" you will get the whole string minus the trailing char.
For now this works for us.
(If not, a fall back I use to test is "msg:string={00-FF}" this will match any sequence of characters in the full HEX range [any character])

To see this work, first click 'activate' on the 'TCP Stream Control' actor, then move to the 'TCP Send Data' actor and click 'trigger'. Why 2 clicks? Remember back to the process sequence, we first need to open a connection to the server, then send the GET request, so that is what we did.

To make things easier, lets add a 'Keyboard Watcher' actor (the default of watching 'a' is fine), and a 'Trigger Delay' actor. Connect the output of the 'Trigger Delay' actor to the 'trigger' input of the 'TCP Send Data' actor, and set the delay time to 0.5 sec
Connect the 'key' output of the 'Keyboard Watcher' to the 'trig' input of the 'Trigger Delay', and to the 'activate of the 'TCP Stream Control' actor.

p1 e

Finished GET request patch

Now when you hit the 'a' key (if you left the default) a trigger will first activate the server connection, then after a short wait, send the GET request. When the server responds you should see the 'TCP In Watcher - Text' actor output a message like "showMyIP({"ip": "99.231.128.74"})" to the 'msg' output, and a trigger to the 'msg rcv' output. This 'msg rcv' trigger then connects to the 'TCP Stream Control' actors 'deactivate' to close the server connection. To run the GET request again, just hit the 'a' key again. You may want to clear the 'msg' output (click it and delete the text) so that you see the text arrive again.

Note: that it will work without the deactivate connection, but this is bad practice, as you should close the connection once you are finished with it, that said you may want to make a number of requests before closing the connection. In these cases it is best to open at the beginning as we have, then make all your communications with the server (GET, POST etc..) and close the connection once all are complete.

An after thought: I noticed that in my images, I have used a 'TCP In Watcher - Text' actor where I have edited the Title to include a note about the text parsing. Please just ignore that ;)