In order to process data (which includes storing data), a data controller must establish one or more of the lawful bases contained at Article 6(1) of the UK GDPR. Briefly, those are:
a) Consent of the data subject
b) Necessary for performance of a contract with the data subject
c) Necessary for compliance with a legal obligation
d) Necessary to protect vital interests of data subject or another person
e) Necessary for public interest or exercise of official authority vested in the controller
f) Necessary for controller's legitimate interests
Clearly a) and d) don't apply. As you've settled the debt, it seems unlikely to me that b) or f) apply. That leaves c) and e). A common example under c) would be a requirement by Companies House or HMRC to keep accounting records for a certain period of time. Some public bodies may also find it necessary, under e), to retain records which they need to be able to carry out their other functions. It seems highly unlikely to me that either of these would justify retention for your "lifetime" however.
I would start by sending them a written demand to have your data erased under Article 17(1). Make sure you also specifically request that they provide you with all the information (and in particular the purposes of the processing) under Article 15(1), and that, in the event that they refuse to erase your data, they explain the reasons why pursuant to Article 12(4). Their response on these points will be helpful in establishing whether there is a lawful basis.
Your next step after that is either to complain to the ICO or to apply to court for a compliance order under Section 167 of the Data Protection Act 2018. The former is free while the latter is not and carries risks of cost if you are unsuccessful. If you opt for one of these steps, make sure you cite the relevant provisions of the GDPR and explain why you think there is no lawful basis for the data retention (including by referencing any response you received from them).
"I have read GDPR guides on how to request erasure, but I don't really feel this applies- I want to have my account deleted, not the track record of the loan and repayment"
It doesn't matter whether we are talking about your account or your track record. What matters is whether the account constitutes personal data, which it almost certainly does, per the definition at Article 4(1):
"‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person;"
