I believe the roving bands of people like you mention, people with kids that are hungry will be easy to handle. A few rounds in their direction will most likely change their thought pattern.
The real trouble will be the looter groups that are armed and will kill you to get what you have. They will be determined and vicious enough to not care who or what is in their way. Their numbers will be overwhelming and eventually you will be overrun.
There are two kinds of people, those that are prepared, and those that suffer.