"And you assume I've not been improper previously?"

Maybe not to grand lengths, sure, but Benedict imagines people would talk if they knew he spent some nights at parties with artists or the general working class of London. Or if they knew that he'd been with a married woman and another women older than him at the same time, or that maintained a frivolous relationship with the older woman for several weeks following. Or the other inclinations that have been coming to mind.

In regards to Benedict's upbringing, that's all quite improper.

"Or that I'm not being improper even now, accompanying you to the docks when I scarcely know you."