In general, a fly will tend to drift with the current (at the same speed). However, currents are rarely uniform between top and bottom or from side to side of the water column due to many factors (bottom structure in the main).
I think the "painfully slow" mantra applies more to and comes from using nymphing techniques. It's important to get down deep near structure where fish hold out of the way of the main current. Not sure swinging a fly will ever be as slow but slowing it down is entirely possible. For example, choose lines that will allow a deep presentation and allow mending as well (sink-tips or sinking heads). Choose water that has a slower current and mend the line as it swings. Cast quartering upstream and across (instead of down and across) allowing the fly to reach depth before the start of the swing and, last, add weight to the leader (split shot) and/or fly (heavier or weighted hooks).
Just some ideas to try....
"America is a country which produces citizens who will cross the ocean to fight for democracy but won't cross the street to vote."