Put a sock(et) in it

If you have a Java program that connects to a kdb+ database then ensure you turn off the Nagle algorithm; which is an optimization algo set by default to ensure not to send payloads smaller than the headers. For instance sending a payload of 1 byte and the TCP headers are 20 bytes would be inefficient so it is best to add more data to the payload before sending. However for latency sensitive applications this efficiency is not desirable. It can be turned off using the following snippet of code:

kx.c connection = ...
c.s.setTcpNoDelay(true)
c.k(".test.c:20j")
c.k(".test.c")

Bingo! Now it will turn off the packet efficiency algo (at the expense of pipe congestion) but it is best to send a payload straight away rather than delaying it. This advice is applicable for any sort of messaging within a Java program with real-time constraints.

Arithmetic overflow in Q

Given the loose typing system of Q, I would have thought that it would promote variables when arithmetic overflow is produced but instead it computes the carry. Come on Arthur, don't be shy, take advantage of your type system and promote its type behind the scenes.

Let me give an example:

q) x:2147483646*2147483646
q) x
4
q)type x
-6h

What it should really do is this:
q) x:2147483646*2147483646
q) x
4611686009837453316j
q)type x
-7h

The only way to do this is explicitly promoting to long before the arithmetic.

q) x:2147483646j*2147483646j
q) x
4611686009837453316j
q)type x
-7h

Which one do you prefer?

Covariance matrices in kdb+

A covariance matrix is at the heart of many data analysis and time series models - as it is a measure of how two data sets change together. In Q, the natural data structure is a vector, which can be nested to represent a matrix. A covariance function is already provided that takes two vectors and computes its covariance, but as a scalar and not as a matrix.

Here is a function I wrote that creates a covariance matrix given a set of vectors. It will create a nxn matrix:

q) covar:{{x[y] cov/: x}[x;] each til (x#:)}

Here is how to use it:

q) a:1+til 10
q) b:10-til 10
q) c:2*1+til 10
q) d:2*1-til 10
q) e:4*1+til 10
q) f:4*1-til 10
q) covar (a;b;c;d;e;f)

8.25 -8.25 16.5 -16.5 33 -33
-8.25 8.25 -16.5 16.5 -33 33
16.5 -16.5 33 -33 66 -66
-16.5 16.5 -33 33 -66 66
33 -33 66 -66 132 -132
-33 33 -66 66 -132 132

Next blog post will be how to compute its eigenvectors and eigenvalue.