Is it possible for him just to pay me my portion of the down payment?
It might be possible, but might not be fair. What would be the most fair is to determine the market value of the house, subtract out the balance of any mortgage, and split the remaining equity 50/50 (or proportionally to whatever you each have paid to principal thus far). If one of you have paid more than 50% of maintenance or other costs, then you might try to get some compensation for that, but there's no law to my knowledge that would require that.
If you do not already have an agreement in place, then I would write one up to protect both of you. If you cannot reach an agreement, and end up pursuing it in the courts, then a judge is most likely going to enforce something very similar.
I am not a lawyer, but this Wisconsin statute would seem to apply (emphasis added):
700.20 Extent of undivided interests in tenancy in common. The extent of the undivided interests of tenants in common for the duration of the tenancy is determined by the intent expressed in the document of title, instrument of transfer or bill of sale; if no intent is expressed in the document, instrument or bill of sale, tenants in common are presumed to own equal undivided interests for the duration of the tenancy.
