Endpoint Data Forms

Article Overview
Data Forms
Templates
Encoding and Data Filters
Where To Go From Here
Notes

Goals

This article covers the various ways that the endpoint API may be used to convert, map or otherwise relate incoming and outgoing data to formats used by external devices. At the end of it you should be able to use these features to effectively translate data to or from a desired data format.

Prerequisites

You should be an experienced NewtonScript programmer who is familiar with the Newton view system and the use of the Newton Toolkit (NTK). It is also better if you have some experience with communications programming whether on the Newton or another platform however this is not strictly necessary. Without experience with communications programming you will be taking on the double task of learning basic communications concepts as well as the Newton communication API. This article assumes that you have a general knowledge of the 2.0 endpoint API. If not, you may want to refer to the article Endpoint Details for a thorough background of this API.

Required Equipment

To write Newton endpoint code you must of course have a Newton and a desktop machine equipped with version 1.5 or later of the Newton Toolkit. You will need an appropriate cable to connect your desktop machine and the Newton for downloading packages from NTK. NTK may be purchased from Apple and the necessary cables can usually be purchased from any store or mail order supplier which sells computer products.

For technical details on endpoints, the "must read" reference is the document: Newton Programmer's Guide: Communications, particularly Chapters 4 and 5 (Endpoints and Built-in Communication Tools). This document is available from the Newton Technical Information web site. This article is based primarily on the information in these chapters. It is probably also a good idea to have a copy of The NewtonScript Programming Language, The Newton Programmer's Guide: System Software 2.0 and Newton Toolkit User's Guide. Finally the Q&A's Newton 2.0 document has several very important descriptions of the interactions between the Newton operating system and an endpoint. All of these documents except the Q&As are available online at the Newton Technical Information web site. The Q&As are available from the Q&A ftp site.

Article Sections

In the Data Forms section we will discuss how data may be transformed when going to or coming from an external device. This covers the use of the form slot in inputSpecs, outputSpecs and option frames.

Templates are a data form used to map NewtonScript values to and from a C language type structure. Templates are used for sending complex option specifications to the low-level communications tools as well as for transferring structured data between the Newton and another platform. Templates are a complex kind of data form.

The Encoding and Data Filters section discusses various ways that data may be converted from one format to another whenever transfer takes place through the endpoint.

Overview

The several ways of transforming data which are discussed in this article are interrelated and can and are often used in conjunction with one another. These relationships are represented in the following diagram:



This diagram shows that incoming and outgoing data of whatever form (including the template), if it contains character data, may go through an encoding process. By default this encoding translates ASCII character values to Unicode values for incoming data and vice versa for outgoing data. In addition, incoming data may also be filtered. This involves mapping byte values or stripping bits as appropriate for the system the Newton is connected to. Finally the form of the data controls how it is translated to or from NewtonScript values. A template is essentially a composite of simple types which allows a series of values to be translated and sent together.

The rest of this article will describe the API and options available for each of these techniques.

Data forms are exactly what the name implies: the form data will take. This specification allows the endpoint system to format them appropriately for the Newton (for incoming data) and in a more or less standard form for outgoing data. For example, if you specify a data form of 'number then the Newton 30-bit value (the upper 30-bits in a 32-bit value) will be placed into the lower 30 bits of a 32-bit value before sending it outwards. Similarly, incoming numbers will have the lower 30 bits put into the upper 30 bits of the Newton destination.

Recall that the lower 2-bits of are used for type information.

Here is a table with the currently defined data forms, all of which are expressed as symbols:

'char - single character immediate value, converted to/from Unicode using the encoding table specified in the endpoint. See the Encoding and Data Filters section for details.




'number - 30-bit integer, converts outgoing by bit-shifting right 2 bits, converts incoming by bit-shifting left 2-bits and adding object specific bits




'string - terminated array of characters, termination character is not sent/received




'bytes - a non-character array of unsigned bytes




'binary - raw data, no conversion is done




'frame - a NewtonScript frame for transfer between Newtons, flattened for outgoing frames, unflattened for incoming frames




'template - complex C struct description of data. See section on Templates below for details.

Data forms appear in three places in the communications system: input specs, output specs and option frames. In an input spec the form slot is set with one of these values to control how data is translated on receipt before it is passed to the InputScript, PartialScript or other input methods. For details on input specs see the Input Spec Animation article. An example of the use of the form slot in an input spec is shown below:

inputSpec := {
form: 'string,
termination: { endSequence:[unicodeCR,"Logout"], },
discardAfter: 256,
InputScript: func(ep, data, terminator, options)
ep:MInput(data),
completionScript: func(ep, options, result)
ep:MNotifyError(result),
}

Here the form slot specifies that the incoming data will be a string. This means that the data will undergo translation from ASCII to Unicode values using the endpoint's encoding slot specified translation table so the data argument to the InputScript will be a NewtonScript string of Unicode characters.

The use is almost identical in an output script. A form slot describes to the endpoint how to translate outgoing data. An example of this is shown below:

outputSpec:={
async: true,
CompletionScript: func(endpoint, options, result)
:MSendCompProc(result),
form:'binary,
target: { offset:0, length:100} // length is ridiculously small to demonstrate target
}

In this case the outgoing data is sent in raw binary form so the outgoing data will be left in untranslated form as it is sent. This is useful for sending such things as packages, pictures, sounds, etc.

In the case of option frames, we are usually controlling the translation of data being sent to low-level communications tools. Since these tools are built in C++, they often expect data in a C struct format. This is done using the Template data form described in the section on Templates below. An example of a simpler data form in an option frame is sent below:

local option := {
type:'option,
label: kCMOSerialBitRate,
opCode:opSetRequired,
form: 'number,
result: nil, // not needed; returned
data: k19200bps, // change to 19200
};

This is the serial data rate option for the built-in serial tool. Here we have specified the data form as being a number and so the NewtonScript integer value for the data slot will be bit-shifted appropriately before it is passed to the serial comm tool.

Templates are a particular kind of data form that is used to represent aggregate data in a form similar to C structures. As mentioned in the previous section they are often used to set options for low-level tools and may also be used to send and receive data from external devices.

Templates are frames with two arrays in them: an arglist and a typelist. The typelist is an array of symbols which describes the data type of each element in the template and the arglist is a list of data values for each element in the typelist. The one exception to this is that the first element in the typelist must be the symbol

'struct since templates are used to represent C structure data. The following table is a list of possible symbols for the typelist array

'long - 4-byte signed integer value



'ulong - 4-byte unsigned integer value



'short - 2-byte integer value



'byte - 1-byte unsigned value



'char - 1 byte character value (translated using Encoding)



'unicodechar - 2-byte Unicode value



'boolean - 1-byte value



'struct - indicates a C type structure, this must be followed by the contents of the structure (see example below)



'array - indicates an array, must be followed by the contents of the array

An example of a template is shown below as a serial option for specifying a connection through the first PCMCIA slot. This template demonstrates an array as an element in the template:

{ label: kCMOSerialHWChipLoc,
type: 'option,
opCode: opSetRequired,
form: 'template,
data: {
arglist: [
kHWLocPCMCIASlot1,
0,
],
typelist: [
'struct,
['array,'char, 4],
'ulong
] } }

If you want to use a template to specify where input will be placed, you must use the target slot to control input. This looks like this:

inputSpec:={
form: 'template,
target:{arglist: [0, 0,"abcd"],
typelist: ['struct,'long,'short,['array,'char, 4]],

InputScript: func(endpoint, data, terminator, options)
begin
fooSlot:=data[0];
barSlot:=data[1];
strSlot:=data[2];
end,
}

Note that the values in the arglist are just placeholders which are replaced with the incoming values. The InputScript is called when enough data has arrived to fill the template and the data argument to the InputScript will be the arglist filled with the real data. For more on the use of the target slot, see pp. 4-40 and 4-41 of NPG: Communications.

As noted above in the section on Data Forms, if the data being transferred is character data (i.e., the form is specified as being 'char or 'string or an element in a template is character data), it is automatically translated from ASCII to Unicode for incoming data and Unicode to ASCII values for outgoing data. The Encoding slot of an endpoint frame specifies a translation table which is used for this mapping.

By default this table is the one specified by the constant kMacRomanEncoding and maps characters according to the Macintosh Roman character set as defined by the Script Manager on the Macintosh. Other values can be specified according to the home character set on the Newton.

An inputSpec may also have a filter slot which describes how characters are altered when they are received and before they are passed to any of the input methods. The filter slot is a frame with a byteProxy slot and a sevenBit slot.

The byteProxy slot is either a single frame or an array of frames. Each frame in the byteProxy describes how to map individual characters. It does this by having a byte slot which identifies single-byte (ASCII) characters and a proxy slot which describes what character it is to be replaced with. A byteProxy frame may have a proxy slot with a value of nil in which case the character identified by the byte slot is removed from the input stream. An example of a byteProxy slot is shown below:

inputSpec:={
form:'string,
InputScript: func(ep,data,termination,options)
begin
// assume something useful
end,
filter: {
byteProxy:[{byte:$00,proxy:nil},
{byte:$0A,proxy:$0D},{byte:$09,proxy:$20}],
sevenBit: true,
},
}

In this example, the byteProxy slot will take incoming string data and remove null (zero) characters from the stream, replace linefeed characters ($0A) with carriage return characters ($0D) and replace horizontal tab characters ($09) with space characters.

The other slot in the filter frame is the sevenBit slot. This slot is a simple boolean which controls whether the high (leftmost) bit of incoming characters is stripped off. If sevenBit is true, then the bit is masked out and becomes a 0. If it is nil or is not present, then the characters are left alone. So in the previous example, because the sevenBit slot is set to true, a character whose hex value was $A5 would change to become $25. This may be necessary as some communications systems, "don't care" about the upper bit and so whether it is set or not is totally random. By stripping the upper bit you will guarantee that the value sent is what you receive.

The article Endpoint Details covers most of the fundamental aspects of endpoint programming. If you need more specifics about endpoint programming this is where we suggest you go next. For more details on specific kinds of endpoints, the article Endpoint Flavors covers how to implement and use specific kinds of endpoints.

The Basic Modem Walkthrough Lab looks at a piece of working sample code going through the steps necessary to define, connect and use an endpoint. It is a good way to get a "code's eye" view of the process. To try your hand at coding an endpoint you may want to start with the Simple Endpoint Lab which establishes a simple serial connection to a desktop machine. For programmers with experience in coding endpoints in the 1.x System, the Converting 1.x To 2.0 Lab is a good hands-on example of the changes between the system.

The Input Spec Animation is a good introduction to input specs and the interaction of encoding and filter forms in input specs.

Of course the detailed documentation on data forms available in Chapter 4 of NPG: Communications is almost required reading. NPG: Communications is available from the Newton Technical Information web site.

Developer Services | Technical Information | Tools | Sample Code

Main | Page One | What's New | Apple Computer, Inc. | Find It | Contact Us | Help

Copyright Apple Computer, Inc.
Maintained online by commscourse@newton.apple.com
Updated 26-Sep-96 by dcd