Oh, don't I know it.
The finder will lock onto the highest frequency direct arrival. This could be the FF's even if they are not the loudest system, depending on where you place your microphone/what the system delays are set to/what frequencies are arriving at the measurement probe first.
The best practice is to start with individual systems and work your way from mains to supporting. The goal is to get all of your systems acting as one (system a+b+c+d+etc=A).
You are hitting on why using dual channel measurement for in-situ performance monitoring is not the appropriate tool for the job, especially if you are at a center FOH type position and the whole system is on.
My workflow is to make initial sound origin measurements from the center line between main hangs. I measure each side side separately (ultimately this can reveal a number of issues with mains), then subs, followed by any fills that may be in coverage of the mains. I'm looking for the "furthest away in time" source. If I'm aligning to backline instruments, whatever is "furthest upstage in time" is aligned to backline, and every other part of the system is delayed back to whatever that loudspeaker subsystem is. Measurement for other uses (voicing, room reflection detection) is done on a one-side-at-a-time basis.
Because of various high pass filters used, sometimes it's not the subsystem one might expect especially if the entire system is not from the same manufacturer or the various subsystems do not use the same DSP settings, HPFs, etc. Nothing like discovering that the attempt to delay subwoofers to mains doesn't work because the main PA is "ahead" of the subs... and I've seen this happen with BEs attempting hasty alignments.
To channel an Ivanism - "It Depends."
So the take-away is that any high pass filters used should be in place *before* making impulse measurements (on which the Smaart delay finder is based) and any voicing EQ should also be roughed in, too.