Data Loading and Computation

Mathematica contains a variety of data loading functions, available through the function Import. This supports many formats such as comma and tab-delimited text data, as well as more specialized formats for graphics, science, sound, and XML. In addition, binary data can be loaded using the function Experimental`BinaryImport. If you find that your particular data is not well supported directly by Mathematica, it is possible that you may use a Java API to load the data, which was described in the previous section.

In order to develop data loading technology, it is probably a good idea to work in interactive Mathematica so that you understand how the data loading functions work. When you have done this, you can add data loading to your web applications. At this point you need to determine the source for your data. The following sections discuss some of the possibilities.

File I/O

If your data files are available to the file system of the computer on which your webMathematica server runs, they can be read with a command like Import. For this to work, the name and location of the data file must be specified. This can be done in several ways. One way involves placing the files into a directory and setting the full pathname of this directory. This suffers from the extreme disadvantage that if you change the name of the directory, you will have to modify all your scripts that use this name. An improvement could be obtained by setting the name of the directory with an initialization parameter. This can be done in MSPConfiguration.xml with the parameter KernelInitializeCode as shown in the following.

<KernelInitializeCode>
MyApplication`DataDirectory="C:\\Work\\Data"
</KernelInitializeCode>

This assigns the Mathematica symbol MyApplication`DataDirectory to "C:\Work\Data". Note the use of a full context name. This is necessary to prevent the symbol being cleared by the kernel cleaning mechanism. This can then be loaded in a webMathematica computation as shown in the following example.

<msp:evaluate>
    data = Import[ ToFileName[ MyApplication`DataDirectory, "file.dat"], "Table"];
</msp:evaluate>

An alternative is to place the data file into a directory that is on the Mathematica path setting $Path. This is the approach taken by the example Data1.jsp, which is shown below. Another alternative is to place the data file into the same directory as the script, and to use MSPPageDirectory, as shown in the following.

<msp:evaluate>
    data = Import[ ToFileName[MSPPageDirectory[], "file.dat"], "Table"];
</msp:evaluate>

This was used in the XML example Phone.jsp, which was discussed previously. It is particularly convenient to use MSPPageDirectory since it means that data and scripts live in the same directory. Thus the entire web application can be moved from one server to another with a minimum of setting up. One disadvantage is that for JSPs the data file can be loaded by a direct request to the server; thus it should only be used if there is no specialized information present in the data file. This might be the case if only certain information was suitable to be used in a response to each request.

HTTP Upload

Another way to load data into a webMathematica server is to send it from the client machine with the HTTP request. webMathematica contains tools to support this with the function MSPGetUploadFile. This is demonstrated in the example, Upload.jsp, which is shown below.

Database Connectivity

DatabaseLink provides Mathematica with an industrial-strength, ready-made solution for integrating Mathematica with any standard SQL database. Integrated since Mathematica 5.1, it provides a convenient bridge between SQL databases and webMathematica.

DatabaseLink is based on Java Database Connectivity (JDBC) technology and so it fits very well with the Java technology on which webMathematica is based. It has many useful and important features (listed at http://reference.wolfram.com/language/DatabaseLink/tutorial/Overview.html). One particularly useful feature for webMathematica is that DatabaseLink contains the HSQL Database Engine (HSQLDB), a lightweight database. This means that if you do not already have a database or want to experiment with using one, you do not have to set one up.

webMathematica contains two examples of working with a database. These are Database.jsp, which is accessible via http://localhost:8080/webMathematica/Examples/Database/Database.jsp and DatabaseTable.jsp, which is accessible via http://localhost:8080/webMathematica/Examples/Database/DatabaseTable.jsp. Note that for these links to work, you need to have installed webMathematica, and also the DatabaseLink example databases, as described in its documentation. The example databases are installed into $UserBaseDirectory, and so you need to make sure this is $UserBaseDirectory for the user who is running webMathematica.

Web Services

Another way to find data for Mathematica is by contacting another website. A particularly convenient way to do this is if the other website provides the data as a web service. In this case Mathematica can use the Web Services Package (described at http://reference.wolfram.com/language/WebServices/tutorial/Overview.html) to call the website and use its services.

Data Examples

The following is a collection of examples of webMathematica working with data.

Loading Data: Load.jsp

If you installed webMathematica as described above, you should be able to connect to this JSP via http://localhost:8080/webMathematica/Examples/Data/Load.jsp. (You may have some other URL for accessing your server.)

This example shows how to load Mathematica packages and how to read a data file that is stored on the server. The data is processed with a data smoothing algorithm and then a couple of plots are made. The source for this page is in webMathematica/Examples/Data/Load.jsp. A section that shows the form tag is shown below.

<form action="Load.jsp" method="post">
<msp:evaluate>
  data = Flatten[N[Import["Data/DataFile1.dat"]]];
  term = 4;
  If[MSPValueQ[$$term], term = MSPToExpression[$$term]];
</msp:evaluate>
Number of smoothing terms:
<input type="text" name="term" size="3" value="<msp:evaluate>MSPValue[$$term, "3"]</msp:evaluate>"/> <br/> 
<input type="image" name="btnSubmit" src="../../Resources/Images/Buttons/evaluate.gif"/>
</form>

<msp:evaluate>
  dataSmooth = MovingAverage[data, term];
  MSPShow[ListPlot[{data, dataSmooth}, Joined -> {False, True}, ImageSize -> 600]]
</msp:evaluate>

In this example, an msp:evaluate tag loads the dataset Data/DataFile1.dat using the Import command. This will search for the data file on the Mathematica $Path and will find the Data directory inside the MSPScripts directory, which by default is located in webMathematica/WEB-INF. It is worth looking inside your webMathematica web application and confirming that you can locate the data file. The example then computes a moving average and plots the original and the smoothed data. This is all placed inside a form element so that it is possible to modify the number of terms.

One weakness of this example is that the data has to be loaded from the data file for every computation. It would be better to save the data somehow. In addition, it might be useful to allow the data to be uploaded from the client. These will be explored in the sections that follow.

Uploading Data: Upload.jsp

If you installed webMathematica as described above, you should be able to connect to this example via http://localhost:8080/webMathematica/Examples/Data/Upload.html. (You may have some other URL for accessing your server.)

This example allows the user to upload a data file and then retrieve the uploaded data. The data is then plotted. The source for this page is webMathematica/Examples/Data/Upload.html and webMathematica/Examples/Data/Upload.jsp. The contents of Upload.html are shown below.

<html>

<head>
<title>Uploading Data</title>
</head>

<body>

<h1>Uploading Data</h1>

<form method="post" enctype="multipart/form-data" action="Upload.jsp">
Enter a file to upload:
<input type="file" size="40" name="file"/> <br/>
<input type="image" src="../../Resources/Images/Buttons/submit.gif"/>
</form>

To generate suitable data,
<a href="DataGenerate.jsp">click here</a>

This example shows how data can be uploaded for processing by webMathematica.

</body>
</html>

It should first be noted that this is an HTML page; it contains no Java or Mathematica inserts. A servlet container will deliver HTML pages just as it delivers JSPs, and they can all be put together in the same directories inside of the webapp, which can also contain other files such as those containing images or movies. This file contains a single form element that is set up to submit a data file using an enctype attribute of multipart/form-data and an input element of type file. When the submit button is clicked, the form will be submitted along with the file to the Upload.jsp. A selection of Upload.jsp is shown below.

<msp:evaluate>
    file = "FileName" /. MSPGetUploadFile[];
    data = Flatten[ N[ Import[ file, "Table"]]];
    term = 4;
</msp:evaluate>

<msp:evaluate>
    If[ StringQ[ file],
        dataSmooth = MovingAverage[ data, term];
        MSPShow[ MultipleListPlot[ data,dataSmooth, 
            PlotJoined ->{False,True},
            SymbolShape -> Point,
            SymbolStyle -> {{PointSize[0.008],Hue[0]}, {PointSize[0.001]}},
            ImageSize -> 600]]]
</msp:evaluate>

In this selection we see how the MSP function MSPGetUploadFile is used to retrieve the name of the data file by which the data has been stored on the server. This filename can then be used to read the data as was done in the previous example and the computation can proceed. It should be noted that this solution is somewhat limited; the data is only available once for one computation, and no facility has been added to change a parameter, for example, to control the data smoothing. For this the data needs to be stored in a session variable. This is explored in the next example.

In this example a test was made to determine whether the filename was in fact a string. This checks that a data file has in fact been uploaded. A file would not be uploaded if the page Upload.jsp was visited directly instead of coming from a request to Upload.html.

MSPGetUploadFile returns a list of useful information, including the filename that is used on the server, the original filename used on the client, and the content-type. In a case where there are multiple files to be uploaded, MSPGetUploadFile will throw an exception. If you wish to upload more than one file, you can use MSPGetUploadFileList.

Session Storage of Data: Session.jsp

If you installed webMathematica as described above, you should be able to connect to this JSP via http://localhost:8080/webMathematica/Examples/Data/Session.html. (You may have some other URL for accessing your server.)

This example allows the user to upload a data file and then retrieve the uploaded data. The data is stored in an HTTP session and then accessed from a separate page. The data is then plotted. The source for this page is webMathematica/Examples/Data/Session.html, webMathematica/Examples/Data/Session.jsp, and webMathematica/Examples/Data/SessionProcess.jsp. The contents of Session.html are shown below.

<html>

<head>
<title>Data in HttpSessions</title>
</head>

<body>

<h1>Data in HttpSessions</h1>

<form method="post" enctype="multipart/form-data" action="Session.jsp">
Enter a file to upload:
<input type="file" size="40" name="file"/> <br/>
<input type="image" src="../../Resources/Images/Buttons/submit.gif"/>
</form>

To generate suitable data,
<a href="DataGenerate.jsp" target="_blank">click here</a>

This example shows how data can be stored in HttpSessions for processing by webMathematica.

</body>
</html>

This is very similar to Upload.html, except that it calls the JSP Session.jsp when the form is submitted. Session.jsp is shown below.

<%@ taglib uri="http://www.wolfram.com/msp" prefix="msp" %>

<msp:evaluate>
    file = "FileName" /. MSPGetUploadFile[];
    data = Flatten[N[Import[file, "Table"]]];
    MSPSessionVariable[UploadedData, Null];
    UploadedData = data;
</msp:evaluate>

<jsp:forward page="SessionProcess.jsp" />

This short JSP reads the data that was uploaded, and then uses MSPSessionVariable to store the data in a session variable named UploadedData. This is a very convenient way to store data persistently from one HTTP request to another. The data is actually stored in the server, so that even if the Mathematica kernel was restarted, the data would still be available. The storage uses what is known as an HTTP session and the server may use cookies or some other mechanism to actually store the data. This is how a shopping cart in an e-commerce website works. As far as a developer of a webMathematica website is concerned, it is all very simple; just use the function MSPSessionVariable, giving it the name of the variable and an initial value. After the data has been read, there is a jsp:forward to the page that actually does the computations and plots, SessionProcess.jsp. Often it is a very good design principle to divide the code over a number of separate pages that all do different tasks; if you develop pages that have large amounts of complicated code in them, then refactoring might improve your system. A selection of SessionProcess.jsp is shown below.

<form action="SessionProcess.jsp" method="post">
Number of smoothing terms:
<input type="text" name="term" size="3" value="<msp:evaluate>MSPValue[$$term, "3"]</msp:evaluate>" /> <br/>
<input type="submit" name="btnSubmit" value="Evaluate"/> 
</form>
<a href="Session.html">Load another data file?</a>
</div>

<div class="section">

<msp:evaluate>
  MSPSessionVariable[UploadedData, Null];
  term = 4;
  If[MSPValueQ[$$term], term = MSPToExpression[$$term]];
</msp:evaluate>

<msp:evaluate>
  If[UploadedData =!= Null,
    dataSmooth = MovingAverage[UploadedData, term];
    MSPShow[ListPlot[{UploadedData, dataSmooth},
      Joined -> {False, True},
      PlotStyle -> {{PointSize[0.008], Hue[0]}, {PointSize[0.001]}},
      ImageSize -> 600
    ]]
  ]
</msp:evaluate>

SessionProcess.jsp is very similar to Load.jsp; the main difference is the line that obtains the data. Instead of reading data from a file, it uses the session variable UploadedData, which was assigned to the data in the previous JSP, Session.jsp. If, by some chance, UploadedData was not previously defined, as might happen if SessionProcess.jsp is visited directly, then UploadedData will have the value Null. Here this just prevents any plot from being produced, but in general it might be used to transfer to an error page.

A further change to this section might be to store the data permanently on the server. For a simple data file, this could easily be done by using Mathematica file output operations. A more sophisticated application could use Java database connectivity to store the data and certain related information.

Session variables are discussed further in Advanced Topics: Variables.

Database Connections: Database.jsp

If you installed webMathematica as described above, you should be able to connect to this JSP via http://localhost:8080/webMathematica/Examples/Database/Database.jsp. (You may have some other URL for accessing your server.)

This example shows how to use DatabaseLink to connect to an SQL database. The database in question is one that is contained in HSQLDB, and for it to work the sample databases need to be installed. This is described in the documentation (http://reference.wolfram.com/language/DatabaseLink/tutorial/Overview.html). The databases are installed into $UserBaseDirectory, and so you need to make sure this is $UserBaseDirectory for the user who is running webMathematica.

This example shows how you can open a connection to a database and make a query to a particular table based on an input parameter. The data is displayed with HTMLTableForm.

<form action="Database.jsp" method="post">
Lower limit: 
<input type="text" name="limit" size="24" value="<msp:evaluate>MSPValue[$$limit,  "6000"]</msp:evaluate>"/> <br/>
<input type="image" name="submitButton" src="../../Resources/Images/Buttons/search.gif"/>
</form>

<msp:evaluate>
  Needs["DatabaseLink`"];
</msp:evaluate>

<msp:evaluate>
limit = 6000;
If[MSPValueQ[$$limit],
  limit = MSPToExpression[$$limit]
];
</msp:evaluate>

<msp:evaluate>
  conn = OpenSQLConnection["publisher"];
  headings = {"TITLE_ID" , "LORANGE", "HIRANGE", "ROYALTY"};
  data = SQLSelect[conn, {"ROYSCHED"}, headings, SQLColumn["LORANGE"] > limit];
</msp:evaluate>

<h3>
Royalty Schedule Table
</h3>

<msp:evaluate>
  HTMLTableForm[data, TableHeadings -> headings, TableAttributes -> {"cellpadding" -> "0", "cellspacing" -> "0"}]
</msp:evaluate>

<!-- 
 The shutdown command is specific to the HSQL database that 
 is used for these examples.
 -->

<msp:evaluate>
  SQLExecute[conn, "SHUTDOWN"];
  CloseSQLConnection[conn];
</msp:evaluate>

The code loads DatabaseLink and carries out some processing of an input variable, which is used to select data from the database. It opens a connection to the database and selects data using the input variable and then formats and prints the result. Finally, it shuts down the database and closes the connection. The shut down instruction is specific to HSQLDB and is necessary to make sure that the database can be used if the server is restarted.