It is likely that you don't owe any taxes even though you have worked for many years in Brazil.
Earned income that is earned outside the U.S. by natural persons who are U.S. citizens (as oppose to entities like corporations) up to a certain amount that is indexed for inflation is excluded from taxation under Section 911 of the Internal Revenue Code. The cut off is $104,100 U.S. per year in 2018 (about 347,000 Brazilian Reals per year).
Also, unearned income and any excess earned income can benefit from the standard deduction and personal exemption from U.S. income taxation unburdened by your foreign earned income up to the threshold. This can be quite substantial. Similarly, if you have children, they would benefit from the child tax credit which would reduce your income tax obligation.
And, any U.S. federal income tax due on your unearned income to the extent that it exceeds the standard deduction and personal exemption from U.S. income taxation is reduced by any Brazilian taxes you paid on your Brazilian income as a result of the foreign tax credit.
The bottom line is that unless you are very affluent and have significant unearned income, it is unlikely that you owe any U.S. income taxes.
Even if you do owe U.S. taxes after these considerations, they wouldn't arrest you. You have to be "willfully" violating U.S. tax laws for it to be a criminal offense and this is one small corner of the law where ignorance of the law is an excuse. And, the Internal Revenue Service would first have to send you several notices by mail setting forth the amount that they think that you owe, allowing you to dispute that amount, and the declaring that you owe it, before you have a potentially criminal tax violation that is well defined enough and communicated to you well enough that you can willfully fail to file a tax return or can willfully fail to pay.
So no, they will not arrest you and make you pay, at least not until they send you lots of notices which you ignore.
On the other hand, if you do owe any U.S. income taxes there is no statute of limitations on collecting those taxes for years in which you were required by U.S. tax law to file a tax return but did not (basically, years in which you owed U.S. taxes). So, if you, for example, won a big lump sum payment playing the lottery when you were 20 years old and owed U.S. income taxes as a result (even though you didn't know it), those taxes could still be collected by the IRS when you are 40 years old since you did not file any U.S. tax return for the year in which you had taxable U.S. income.
Finally, under the Tax Cuts and Jobs Act of 2017 enacted in December last year, U.S. taxation of the foreign income of U.S. persons was reformed in the most significant way in the last seventy years or so, and the taxation of U.S. persons on foreign source income is greatly curtailed for 2018 going forward. This is not retroactive, but could mean that as you come into your higher income earning years in middle age, that the taxation of U.S. citizens on their worldwide income is no longer a serious concern for you, even if it would have been under U.S. tax law before then.
Enforcing laws that have since been repealed is also rarely an administrative priority for the IRS. It is also hard (to the point of being economically not worth it) to enforce U.S. tax laws against someone who has no U.S. assets.
It is also possible that even though you were a dual citizen at birth, that you could have taken an act (e.g. voluntarily serving as an officer in the Brazilian military) that could cause you to lose your U.S. citizenship. So, even if you were a U.S. citizen at birth, it is not 100% certain that you are a U.S. citizen now.