Using the HTML5 draggable api should not be difficult. This makes it simple to swap elements by dragging.
Toggle draggables