Thanks everyone for the recommendation so far
At the moment my Plan A is to drive from CHC - Kaokorua. Following day drive up and drop the vehicle at Picton before boarding the ferry.
OR
Plan B drive back CHC the following day and drop the car at CHC Airport before taking domestic flight back to WLG.
I understand the Interislander ferry trip can be really nice. Not sure if we'd be missing much if we've done the Milford Ferry (also 3hrs long). Alternatively maybe I should just enjoy the view of Southern Alps by taking domestic flight from CHC to WLG