Yes, I have to say, the fact that it is not "one more red brick building" is definitely in it's favor. The pictures you listed are from the 12th street side of the building not the Broadway side. Yes, that portion of the building should be saved, that portion has historic value. It is the portion along Broadway that I think should be revisited.
Regarding the set back, my concern is that at present it is a parking lot, a very suburban model. It creates an unwelcome void in the street scape. Yes, urban park spaces between buildings can be quite an Oasis. If the wall separating this space from Broadway were removed, and a cluster of restaurants and retail were able to open up on to a public plaza with lots of green and flowing water elements, then yes, I am sold on the set back. However to keep it as a parking lot walled off from the most pedestrian friendly street in Columbus, that to me is not desirable.
Yes it was wrong to demolish the old courthouse just as it was wrong to demolish the old city market in Savannah to build a parking garage. Those mistakes and others led to the Historic Preservation movement. However, the desire to protect the valued sometimes is stretched to protect all that is simply older than the new. To me, a smarter approach is to view the city much as a gardener views his garden. Selectively planting, pruning, and weeding as necessary to create something beautiful and enjoyable.