The Gospels repeatedly place tax collectors beside people regarded as sinners and social outsiders. That pairing reflects more than dislike of paying taxes. Tax collection operated within structures of political domination and could create opportunities for abuse and personal enrichment.
Jesus' willingness to eat with tax collectors, call one as a disciple, and use their reputation in his teaching therefore carried substantial social force.
Taxes and tolls were collected through administrative arrangements in which local agents could stand between the population and ruling authorities. Such collectors were easily associated with foreign or client-rule power, and the system could foster suspicion that collectors profited at the public's expense.
The exact arrangements differed by tax and period, so it is better not to imagine every Gospel tax collector performing precisely the same office. What is consistent in the narratives is their poor social reputation and their association with people considered morally suspect.
This context explains why meals with tax collectors provoke criticism. Table fellowship could communicate acceptance and social relationship, so Jesus' conduct was more striking than a casual conversation with an unpopular government employee.
It also gives force to passages in which "tax collectors" represent people assumed to be outside respectable religious society. Jesus repeatedly overturns the expectation that such reputations place a person beyond repentance, discipleship, or God's mercy.