I don’t think a wrap dress in its very essence is non-formal.
In fact, in searching for visual examples I did come across several images of things that are technically wrap dresses but do come across as at least semi-formal.
So it’s not the wrap part that makes it casual, I think it’s just that the typical execution of a wrap dress includes less party-formal details like collars, buttons, 3/4 sleeves, the aforementioned jersey fabric, etc.
This is one of the areas where I think the scale of party-formality (for women) departs significantly from business-formality. If I had to guess why, it would be that business dress takes many more cues from menswear than traditional social occasion dress.
I would actually consider a skirt rather than a dress, because a more formal-looking skirt can be dressed down using the top. They can’t be mismatched too much, of course (a t-shirt and a very formal skirt say “trashy” not “dressed down”), but it may open up your options a lot more.