If God did not care what people wore to church, he would not have had his apostles give instructions on it.
Women should adorn themselves in respectable apparel, with modesty and self-control, not with braided hair and gold or pearls or costly attire, but with what is proper for women who profess godliness—with good works. (1 Tim. 2:9-10)
Apparel should be "respectable." a word meaning decent or well-ordered. We can infer from this, not that we ought to dress semi-formally and not casually, but that what we
do wear should be neat, clean, and suitable for the occasion. God doesn't demand a suit and tie. But we shouldn't come to his worship looking like we slept in our clothes, either. Take some effort.
Apparel should be modest, which here means "less extravagant," as modest apparel is contrasted with apparel adorned with gold and jewelry. We don't need to show off our status by our accessories or expensive hairstyles. Good works fit for a Christian are decoration enough (v. 10).
Paul addresses women, probably because they tend to be more inclined to showy outfits. But clothing is a status symbol for both men and women. Men don't tend to buy expensive outfits that they only wear a few times, but we don't need to come to church in tailored suits or shirts with designer logos on them, either.
In summary, God doesn't care if you dress casually for church. He does care that you come dressed neatly and respectfully and not to draw attention to yourself.