The phrase is typically used to refer to babysitting, i.e., I’ll make sure the kids don’t leave or kill themselves. So the “down” serves a purpose in the present use, regardless of its applicability to historic situations.

“Holding the fort” just meant not letting enemies in

“Holding down the fort” also includes preventing things from getting out of hand

But I like the bouncy castle imagery as it’s less mundane