Duke Jai was a Han dynasty (汉朝) government official.
Because he held a very high-ranking and powerful position, many of his friends, relatives, and even acquaintances whom he barely knew often went to call on him. So all day long, horses and chariots were lined up in front of his house; it was really as if "the courtyard was as crowded as a marketplace."
Later, however, Duke Jai had the sad misfortune of being removed from office.
His friends and relatives then stopped going to see him, and soon the only ones left in front of his house were flock of sparrows which would fly about and stop to rest on his doorstep.
Not long afterwards, Duke Jai was reinstated. As soon as his friends and relatives heard the news, they all once again began to ride their horses or drive their chariots back to visit him.
Duke Jai, however, was now unwilling to see them, and door reproving those who would only associate with people of wealth and status.
Today, we can use this idiom to describe any place which has been deserted, or where people are few.