You can go Hanoi or Ho Chi Minh City, I dun know much about Hanoi so just say about HCMC.
Normally no need to book for hotels, just go to Ben Thanh market, walk around and get one hotel. but u got a cute child so I think better book in advance for free mind, can try this one:
http://www.ananhotel.com/ (25 USD per night), cheaper one:
http://www.lucysaigon.com (16 - 20USD per night), cheapest ones try
http://www.hostelbookers.com, look for the one near Ben Thanh market. The 2 hotels above I often recommend for my friends when they go to Vietnam. First one nearer to Ben Thanh Market than the second one. Btw, high-class hotels can find here:
http://traveltovietnam.com/
For a relax trip and not-much-shopping, I think just stay in HCMC first day, first night, second day, then take a tour to somewhere else. Can try Vung Tau beach (2 hours going by bus) as saigonblack said above, its near to HCMC. Nicer but further place is Mui Ne beach (Phan Thiet City) (4 hours going by bus). If dun want beach, then can try Da Lat City (4 hours by bus), chill and peaceful place, but this one, normal tours take 4D3N, so I think u should plan 6D5N in Vietnam. No more place to recommend within ur time budget.
For booking tours in Vietnam, can try: http://www.sinhcafevn.comundefined/ (my friends often book here)
if wanna go both HCMC and Hanoi, can try book from Sin to HCMC, enjoy HCMC, fly to Hanoi (can go by train but takes 1 or 2 days to Hanoi), then fly back to to Sin. I think should have at least 5 nights in Vietnam. Btw, Hanoi is turning cold this season, so prepare clothes if u plan to go there. Same if you go Da Lat.
currently Sing Air got promotion $198 + $124 [tax] for a return ticket ( 2 persons go same days ), which period I am not very sure, can ask them.
Hope it helps. And enjoy! Btw I going back vietnam from 25 Oct to 5 Nov. If u guys go there this time, I am willing to help if needed. Dun worry I do not charge haha.