Notice that in order to obtain the minimum swaps, we should focus on the smallest element that is the leftmost and the largest element that is the rightmost.

We can take those elements and greedily only do swaps that bring them closer to their respective end positions.