There's several different parts to this.
- Is it legal to reverse engineer the program
- Is it legal to reverse engineer the format (from files)
- Is it legal to distribute software that consumes the proprietary format
- Can you be sued for any of this
For #1, EFF has a good FAQ for the US.
Acts in the UK up to 2020 (Brexit) would be covered under EU Directive 2009/24/EC, which makes reverse engineering for compatibility purposes legal in many circumstances (including if you don't have the author's permission to use the software). This article has a good overview.
For #2 it's important to know that the output of a program is not copyrighted by the author of a program, although it might be copyrighted by the user who operated the program.
So you should have permission to hold a copy of the files you're inspecting if they contain copyrightable elements (for example, the text inside a .doc file belongs to the person that wrote it).
#3 could be illegal if there's a software patent.
Famously the distribution of software that read GIF files was hampered by a patent on the LZW compression method. Another famous example is Microsoft's FAT filesystem, which resulted in lawsuits to companies reading SD cards.
In some of these cases people found restricted ways of reading the formats that were not covered by the patents.
This is extremely dependent on your jurisdiction. You might want to consider publishing your program from a friendly jurisdiction to avoid legal problems
#4 is the easiest to answer. You can be sued by anyone for any reason at any time.
Lawsuits involving proprietary formats are usually brought against commercial use of the format (as in the FAT case).
If you're working on open source and a company threatens to sue you and you're confident what you're doing is legal you might want to consider asking for legal assistance from the EFF