Deliberate choices are necessary to achieve a cohesive look. This even means that you can wear a more formal looking spread collar shirt (button cuffs) if most of the other pieces of your ensemble say “casual”.
One of the elements in a well put together casual look is the patch pocket jacket. Usually make of more textured (and heartier) cloth than you standard suit jacket, no one will ever think that you are simply wearing the top half of a suit. You can choose whether to wear a tie or show a bit of neck but in either case don’t forget the pocket square.