go hard with (someone)
To be difficult, problematic, or unpleasant for someone; to be to someone's disadvantage, harm, or misfortune. It will go hard with John if he doesn't realize that Tommy is only interested in him for his money. It's going to go hard with us if the government cuts our funding any further.
go hard with
Fare ill, be to one's harm or disadvantage. For example, If this case gets to a jury, it will go hard with the defendant. [First half of 1500s]