Back in July I did this hike from the river, up to the swimming hole at the backpacker campground called Hermit Creek, then up to the historical site above the campground, and looped on the cutoff trail back down to the river. I was on a 17-day, 280-mile private river rafting trip. We had a layover day at Hermit, camping there two nights. This also allowed for passenger exchange via the Hermit Trail. 5 of our group hiked out, and 6 new ones hiked in.
No one from my river running group opted to go on the hike with me, so they missed out on the swim. It's hard to find that swimming hole, since it is not on the trail. If you are coming up from the river you have to leave the trail and bushwhack up the creek once the trail starts up the left-hand cliff. Or, follow the trail, and then loop back down to the campground. The waterfall and swimming hole are just below the campground, right below one of the campsites, in fact. I had been there a few times before, but only as a backpacker, from the rim.
It was a hot day in July, so I wore shorts, Tevas, a big hat, and a long-sleeved cotton dress shirt, which I kept completely wet by dipping myself in the creek during the hike. So, I never overheated at all. It was a beautiful day, but there were no backpackers in the campground at all. A few showed up at the river later on that evening, however.