I just finished adding support for Famicom/NES trackballs to the Retro Adapter firmware. I can now use it as a mouse :-)
The protocol is quite simple. Data is read from the controller in the same way as a normal gamepad. The first 8 bits are the same, followed by 4 bits of y axis offset and 4 bits of x axis offset. Finally there is an ID nibble of 1000. The axis data is in two’s complement format, most likely just the output from some up/down counters.
I opened the trackball up for a look and it uses a custom Hori controller IC labelled “TRK-81-R01″ and “BU3219 844 315″. The only other components are two optical encoders for the ball, two switches (high/low speed, left/right handed), a few resistors, a single power supply capacitor and a 4093 Schmitt trigger. The two ICs are both DIP format. The PCB is single sided and does not use any jumper wires; however there is a patch wire which corrects a off-by-one-pin wiring error. The whole thing could be done in pure logic ICs but Hori must have thought it was cheaper to use a custom IC. Considering the age and limited popularity of the device that’s somewhat surprising.
Unfortunately the deletionists destroyed the Wikipedia article on Hori.