Vectorized Operations

Array-based languages generalize operators and functions to work transparently over multidimensional arrays as well as scalars, thus providing a higher-level conceptualization of data manipulation than scalar-based languages (which require loops to iterate over all the individual members of the arrays).

The following XJ function computes the usual arithmetic average:

avg =: sum div num

Note that there are no references to the parammeter names anywhere in the code: this style of function definition is called function-level programming. This avg will work for any number and kind (as long as they're specializations of numeric) of arguments.

A more familiar implementation follows, using functional programming:

avg values is (sum values) div (num values)

Finally, an implementation using imperative programming:

avg values is

sum: 0

for v in values do

sum: sum + v

end

n: num values

return sum div n

)

These examples clearly point out the significant reduction in code size (mostly due to spurious complexity) that can be achieved with the vectorized operations available in VectorSQL and XJ. Vectorized operations are also often optimized in ways that a function operating on successive scalars inside a loop could not be subject to, resulting in significant performance gains.

As a complete example, here is a very concise library of commonly used statistical functions written in J (NB. introduces a comment):

mean =: sum div num NB. sum div by cardinality

norm =: diff mean NB. differences between values and mean

var =: mean @ sqr @ norm NB. mean of square of norms

stdev =: sqrt @ var NB. square root of var

cov =: mean @ (* & norm) NB. mean of (values times norms)

corr =: cov div (* & stdev) NB. cov div by (values times stdevs)

It is instructive to workout the same code on a traditional scalar programming language for comparison.