Sorry went back and read the first post and noticed you're testing with a 2626. I can reliably run mine at 2x64 buffers in SAC packed out with 24 channels of processing on an IBM 2GHz Core Duo laptop. Round trip is about 7ms. 1x64 should be workable if the MultiRack code is halfway decent.
Another suggestion from the SAC world. Find and download dpc latency checker. It'll measure background processing latency for your system. If you see spikes into the yellow or red, then you've got stuff stepping on the CPU in the background. Wireless networking, unused network cards with no cable plugged in, APCI battery management, all kinds of things that could be causing problems. I had a Toshiba laptop that couldn't run SAC reliably unless I disabled every piece of unused hardware, all networking features and it still gave me occasional problems. From experiences on the SAC forums, some systems will never play nicely. Graphics cards can also be problematic. If you're using a laptop you're SOL. Otherwise you can try swapping out the card. Most find ATI cards are less problematic than NVidia, but there are plenty of people using NVidia cards in SAC rigs as well.
I'm guessing you're running Windows. If so, another item to look for online are the BlackViper guides to streamlining Windows. Basically you turn off all the flashy UI features and disable all unnecessary services.
The biggest lesson was that multiple cores are not a help in realtime processing. With current hardware architectures, only one core can access the same bit of RAM at any one time. You might try booting with just one core enabled if you can.