Whether the rumors are true or not. As long as you know of the facts of radiation, one should decide whether to go or not. Anyway, I for one will not hesitate to go there whether in Tokyo, Osaka or Hokkaido.
Effects from radiations are usually long term, unless one is exposed to a sudden large dosage. You won't know until years or even decades down the road. So if someone tells you they feel it, it is bull. You wont notice it even if you travel to Fukushima restricted area. Unless you bring a calibrated radiation meter or radiation badge, which you will probably be stopped at Jap airport, unless you have valid reasons to bring those in.
Mutational effects from radiation varies from individual to individual and will only manifest itself years down the road, and even if one developed cancer, there is no way to proof that it is because of a certain episode (providing you have controls in the group).
Young female will be more affected than male should they want to give birth afterwards as the egg cells in their ovaries are already there. Therefore, having slightly more effects when they are exposed to a higher dosage of radiation.
The doc suggestion dont give birth in 3 years is also bull as I said earlier, the egg cells are fixed and already there in the ovaries, that slowly develops and mature monthly for the rest of the reproductive age of a female. So should the cell is damaged, it will be there until reaching maturation. However, no effects to sperm as they are made new.
If you are going there for a short period of time, just treat it as a few dosage of X-ray. I think the locals should be more worried than tourists as they have no choice but to remain in the area.
And lastly, if one has doubt and afraid of potential contaminant, it is suggested that one should not go. The psychological harm far out-weight true physical harm.