Jesus has the hardest time with "religious" people. Religion is about being obedient or performing certain rituals or avoiding certain behaviors so as to make myself acceptable to God. The Pharisees had developed such a "hard line" definition of the Sabbath that they even forbade doing good things (such as healing the distressed man in the story) on the Sabbath. The Pharisees are so upset when Jesus heals the man on the Sabbath they collaborate with the Herodians (verse 6) on how to kill Jesus. A common enemy (in this case Jesus) makes for strange bedfellows. The Pharisees were the hyper religious types, the Herodians were political and collaborators with the occupying Roman government.
Notice, also, the opposition from the religious authorities (the scribes) -- 1st century Bible scholars -- and from his own family. Jesus mother and brothers are distressed about his activities and reputation. The brothers mentioned, the younger children of Mary and Joseph, are listed by name in Matthew 13:55. The oldest of whom is known to the early church as James the Just is also the likely author of the epistle of James and presides at the Jerusalem council in Acts 15. Jude (or Judas) is the likely author of the epistle of Jude. It is worth noting that although Jesus mother and brothers do not understand his mission and purpose at this early point of his ministry by resurrection day they are part of the early Christian movement. Jesus mother is at the foot of the cross and is in the upper room on Pentecost and James is visited by the Risen Jesus (see 1 Corinthians 15:3-7).
Followers of Jesus are not worried about what they have to do to earn God's favor, followers of Jesus know that God, through Jesus, has already made us acceptable. We strive to live into that grace filled, grace driven, grace provided relationship.