I believe it is more crowded between Fri and Mon.
Do you want to consider between Tues to Thurs?
There's a crowd calendar that you might want to use for reference?
Not sure how accurate it is though (
https://tdrexplorer.com/universal-studios-japan-crowd-calendar/). There's a graphical guide and a link to the "crowd calendar" in that webpage above.
Super Nintendo World can take up to whole day (with the queues) if you want to partake in all their mini-games/activities and rides (most of them would require the use of the Power-up Band for a better and immersive experience). My family spent pretty much the 3/4 of day there at SNW on 1st day, and half a day on 2nd day.
Harry Potter wasn't that crowded when I went in the evening (as per my allocated timed entry from my express pass), but there wasn't anyone checking our timed entry tickets (goes to show that it wasn't crowded hence no enforcement).
Do take note of the
height limit (
https://www.usj.co.jp/web/en/us/service-guide/safe) for the rides, which your 4 year old might be constrained (just my assumption for his/her height). Something to consider when buying the express pass for your 4 year old.
Do note about the "
Child Switch" (
https://www.usj.co.jp/web/en/us/attractions/how-to-fun#child-switch) system that USJ have that allows parents to take turn on the rides when you have a young one who is not eligible for.
Drawing my example, my 3 year old was not eligible (height) to ride the Mario Kart, so my wife and I had to take turns (with 1 parent staying back with my 3 year old at their waiting room). Good thing is that my 5 year old get to ride Mario Kart twice consecutively with each parent (express pass only allows 1 usage of express queue for designated ride). Bad thing, my 3 year old can't really enjoy (and does get grumpy) when we go for these rides (something to note).