When Joseph’s brothers descended to Egypt to purchase food, Joseph treated them harshly. The Sages describe the background to these events.

“Joseph’s brothers went down” (Genesis 42:3). The verse should have said: The sons of Israel, or: The sons of Jacob. Initially they did not treat Joseph like a brother, and sold him, and ultimately, they regretted it and said: When will we have the opportunity to descend to Egypt and bring our brother back to our father? When their father told them to descend to Egypt to purchase food, they all agreed to bring him back. Rabbi Yehuda bar Shimon said: Joseph also knew that his brothers would be descending to Egypt to purchase food. What did he do? He positioned sentries at all the entrances to Egypt and told them: See everyone who enters to purchase food, write his name and the name of his father, and bring me the lists. They did so. Once the sons of Jacob arrived, they each entered a different gate and the sentries wrote their names. In the evening, they brought Joseph the lists. This one read: Reuben son of Jacob, and one read: Simeon son of Jacob, and one: Levi, and likewise all the gatekeepers, each wrote his note with a name of one of the sons of Jacob. Immediately, Joseph told them: Seal the storehouses and leave open one storehouse so that they will be forced to go there. Joseph gave the names of his brothers to the owner of the storehouse, and told him: When these people approach you, seize them and send them to me. Immediately when the brothers appeared before Joseph, “he acted as a stranger to them and spoke harshly to them” (42:7). The verse teaches that he acted like a stranger to them. Joseph took his goblet and knocked on it as if to perform divination. He said to them: I see in the goblet that you are spies, which is why you entered through different gates. They said to him: “We are sincere” (42:11), but so our father commanded us: Do not enter in one gate.(Bereshit Rabba 91)