Example of a check with no exact amount but with an upper bound by indicating a "Not To Exceed X USD" on the memo line of the check:
If the person who cashed out the check goes over the limit, can I successfully dispute it?
The above picture comes from Los Angeles County clerk (mirror).
Assuming UCC ยง 3-407 applies in the the relevant jurisdiction:
The recipient of an "incomplete instrument" (i.e., a check without the numbers filled in) cannot fill in those details without permission from the issuer of that check. Such a change would constitute an "unauthorized addition of words or numbers [...] to an incomplete instrument." However, such changes are permitted if the change is authorized (both because it would no longer be unauthorized per 3-407-a and because the change would have "assent" per 3-407-b).
In other words, you can use an "NTE" memo on a check, but you'll need to ensure that the recipient of the check is aware that the NTE line constitutes authorization to fill in the value of the check (and what value is authorized). In the case of a county clerk who explicitly instructs issuers to use such a memo line, the use of the memo line likely constitutes authorization for the clerk to fill in the blank values in accordance with the clerk's documented policies.
Note that none of this necessarily binds the bank. A bank employee who notices and understands that the NTE line was violated should reject such a check, since the employee has notice of the alteration as described in 3-407-c . However, more often than not the bank will ignore the memo line; I don't believe the bank has any obligation to notice such content.
I submit that this might work at some California courts, but only to instruct the clerk how much you are willing to spend. Do not expect a bank to look at the memo line. This seems to be a way for the court to have you submit a check for something for which the final charge isn't yet known (like a per page charge for a document that you don't know the length of).
There are cases where a memo can be used to effect a legal stance (Paid in full notes in some states), but you would be charting new waters trying to dispute a check you gave out without an amount, except where is is specifically defined how it will work, like in your example.
