Assuming weather is ok, you are correct to plan the visit to the
Shibasakura (pink moss) festival in the afternoon.
The location of the pink moss area is just south of Lake Motosu with Fuji to the east. Which means that if you visit in the morning, your photos of Fuji will likely suffer from back light with the sun behind it.
By visiting in the afternoon, you will get good light from the western sun shining on Fuji (and the pink moss in front of it) as shown in my earlier second photo. But try not to be there too late (it's a long drive from Kawaguchiko) as the foothills to the west will cast shadows on the flowers if the sun has set too low!
The pink moss and phlox will likely be in full bloom when you are there so your photos should be even nicer than those I took when I was there, weather permitting.
And if you have your own transport, that area is pretty near to the famous Lake Motosu view of Fuji in the 1000 yen note: