Calling the API
Back to the Data Import API main page
Updating or appending data to a SpatialKey dataset using the Data Import API is a multistep process. Each step involves executing an HTTP call to a specific end point and passing specific URL or POST attributes. The three (possibly two) steps of uploading data through the API are:
- Determine the correct cluster URL.
- Login to the server
- Upload the data
Getting the Cluster URL
A SpatialKey organization exists on a server cluster. It is important that calls to the API are routed to the correct cluster or there will be problems with the upload. Each organization has a SpatialKey domain assigned to it and that organization domain can be used to find the correct server cluster URL. For example click or paste the following URL into a browser: http://um.spatialkey.com/clusterlookup.cfm
This link will return an XML document containing information about the organization. Try a similar URL replacing um.spatialkey.com with your organization’s domain (http://myorgname.spatialkey.com).
Figure 1: Return XML example from cluster lookup call
<?xml version="1.0" encoding="UTF-8"?> <organization> <error/> <cluster>cluster2.spatialkey.com</cluster> <organizationID>1</organizationID> <protocol>https://</protocol> </organization>
The XML can be used to prepare for followup API calls as the following code demonstrates. This code and a full sample application (written in Java) is available here that demonstrates all of the topics presented in this document. You can download and use this code as needed (it is provided Open Source).
Figure 2: Cluster lookup code
try {
//get data as XML
URL url = new URL("http://" + skOrganization + "/clusterlookup.cfm");
SAXReader reader = new SAXReader();
xml = reader.read(url);
} catch (Exception e) {
logger.error("[lookupClusterURL] Lookup of domain: " + skOrgDomain + " returned an error: " +
e.getMessage());
return;
}
Logging In to the Server
With the server cluster URL in hand, we can now call the other methods of the API. First we must authenticate into SpatialKey. In order to authenticate the API requires the following: a SpatialKey user name, a valid password for the user, and the organizations prefix (if the organization is um.spatialkey.com, the prefix is um). A call needs to be made to the cluster url (using HTTPS) passing along the appropriate information. Here is an example of what that HTTP call could look like:
Figure 3: Authentication URL example
https://[returned cluster domain]/SpatialKeyFramework/dataImportAPI?action=login&orgName=um&
user=test@spatialkey.com&password=12345
See the Login action API definition for more information.
The HTTP return from this call will include two important pieces of information, the HTTP status code and the server session’s JSESSIONID in the return header. If the HTTP status is 200 then the authentication was a success. All calls to the server after a successfull login need to have the returned JSESSIONID passed along as a cookie. This makes sure that each call stays within the authenticated scope of the login. Here is a Java example of extracting the JSESSIONID and using it in a subsequent call:
Figure 4: Login execution and gathering of the JSESSIONID
// Execute login
try
{
int result = httpclient.executeMethod(get);
if (result != HttpStatus.SC_OK)
throw new Exception("Login failed: " + get.getStatusLine());
String response = get.getResponseBodyAsString();
logger.debug("["+ taskName + " - " + taskType + "] [login] server response: " + response);
//get the headers (need to find the JSESSIONID)
Header[] headers = get.getResponseHeaders("Set-Cookie");
boolean foundId = false;
for (Header header : headers)
{
//set cookie headers have multiple values, need to get the JSESSIONID one
if (header.toString().contains("JSESSIONID="))
{
jSessionID = header.getValue();
foundId = true;
}
}
if (! foundId)
throw new Exception("No JSESSIONID found in returned headers.");
}
finally
{
get.releaseConnection();
}
Figure 5: Using the JSESSIONID on another call
// Get file to be posted (uploading the .zip containing the .csv and .xml)
File input = new File(outputDirectory + processingId + ".zip");
// Prepare HTTP post
PostMethod post = new PostMethod(strURL);
post.getParams().setBooleanParameter(HttpMethodParams.USE_EXPECT_CONTINUE, true);
post.getParams().setSoTimeout(10000);
//create the file part
Part[] parts = {new FilePart(processingId + ".zip", input)};
// Request content will be retrieved directly
// from the input stream
post.setRequestEntity(new MultipartRequestEntity(parts, post.getParams()));
// set the JSESSIONID cookie - VERY IMPORTANT, needed so that the logged in user's session is used!
post.setRequestHeader("Cookie", jSessionID);
// Get HTTP client
HttpClient httpclient = new HttpClient();
Zipping up the Files
Although it is not required, it is recommended to ZIP the CSV and XML files up into .zip file. This will reduce the overall size of the CSV (as they can grow quite large) and therefore cut down on network traffic as well. The process for zipping files up via code is usually fairly easy. In the example Java code we are using the java.util.ZipOutputStream class for this purpose.
Figure 6: Zipping the CSV and XML files
private void zipPackage() {
byte[] buffer = new byte[18024];
try{
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(outputDirectory + processingId +
".zip"));
out.setLevel(Deflater.BEST_COMPRESSION);
// .csv file
String csvFile = processingId + ".csv";
FileInputStream in = new FileInputStream(outputDirectory + csvFile);
out.putNextEntry(new ZipEntry(csvFile));
int len;
while ((len = in.read(buffer)) > 0){
out.write(buffer, 0, len);
}
out.closeEntry();
in.close();
// .xml file
String xmlFile = processingId + ".xml";
in = new FileInputStream(outputDirectory + xmlFile);
out.putNextEntry(new ZipEntry(xmlFile));
while ((len = in.read(buffer)) > 0){
out.write(buffer, 0, len);
}
out.closeEntry();
in.close();
out.close();
logger.debug("["+ taskName + " - " + taskType + "] [zipPackage] '" + outputDirectory +
processingId + ".zip' was created.");
} catch (Exception e) {
logger.error("["+ taskName + " - " + taskType + "] [zipPackage] Error!",e);
}
}
Uploading the Data
The upload itself required only one more server call. The .zip (or CSV and XML) file needs to be placed as a multi-part form post HTTP call. Once again, the process for doing this are programming language specific. In addition to attaching the files to the form post, additional arguments are required as well. It is at this point that some decisions need to be made about the type of upload needed for this dataset. Is the data appending or overwriting data? Should the API return as soon as the upload is finished or after the data is processed and entered into the SpatialKey database? Should an email be sent by SpatialKey when the import is complete? Each of these questions is answered with a URL argument:
Figure 7: Upload URL example
https://[returned cluster domain]/SpatialKeyFramework/dataImportAPI?action=overwrite&
runAsBackground=true¬ifyByEmail=false
See the Overwrite or Append action API definition for more information.
Here is the example applications code for executing the upload:
Figure 8: Upload data example
private String uploadZip(boolean overwrite) {
logger.info("["+ taskName + " - " + taskType + "] [uploadZip] Enter");
String ret = "";
try {
//check skServiceURL, look up if needed (will call a service to get the cluster url) -
//cached after first call
lookupClusterURL();
// Get target URL (see API documentation for URL information)
logger.info("["+ taskName + " - " + taskType + "] [uploadZip] About to connect to: " +
skServiceURL + "/SpatialKeyFramework/dataImportAPI?action=" +
(overwrite ? "overwrite" : "append"));
String strURL = skServiceURL + "/SpatialKeyFramework/dataImportAPI?action=" +
(overwrite ? "overwrite" : "append") +
"&runAsBackground=" + (runAsBackground ? "true" : "false") +
"¬ifyByEmail=" + (notifyByEmail ? "true" : "false");
// Get file to be posted (uploading the .zip containing the .csv and .xml)
File input = new File(outputDirectory + processingId + ".zip");
// Prepare HTTP post
PostMethod post = new PostMethod(strURL);
post.getParams().setBooleanParameter(HttpMethodParams.USE_EXPECT_CONTINUE, true);
post.getParams().setSoTimeout(10000);
//create the file part
Part[] parts = {new FilePart(processingId + ".zip", input)};
// Request content will be retrieved directly
// from the input stream
post.setRequestEntity(new MultipartRequestEntity(parts, post.getParams()));
// set the JSESSIONID cookie - VERY IMPORTANT, needed so the logged in user's session is used!
post.setRequestHeader("Cookie", jSessionID);
// Get HTTP client
HttpClient httpclient = new HttpClient();
// Execute request
try {
logger.info("["+ taskName + " - " + taskType + "] [uploadZip] Sending...");
int result = httpclient.executeMethod(post);
// Display response
logger.info("["+ taskName + " - " + taskType + "] [uploadZip] Server response '" +
post.getResponseBodyAsString() + "'");
ret = post.getResponseBodyAsString();
if (result != HttpStatus.SC_OK)
throw new Exception("Upload failed: " + post.getStatusLine());
} finally {
// Release current connection to the connection pool once you are done
post.releaseConnection();
}
} catch (Exception e) {
logger.error("["+ taskName + " - " + taskType + "] [uploadZip] Error!",e);
return "ERROR";
}
logger.info("["+ taskName + " - " + taskType + "] [uploadZip] Exit");
return ret;
}
No other information is needed as the user is still logged in and the dataset information is provided for by the XML file. Once again an HTTP status of 200 means that the upload was successfull.
The user is automatically logged out at the end of the call, so any additional uploads should start with an authentication again.
Follow Us: