The form calculat is late Latin, and so we may not find this exact phrasing in our earliest Roman sources. The general principle regarding correction of numerical errors is in the Digest of Justinian (530-533), which deals in 49.8 with decisions which can be rescinded without an appeal. The authority cited is Aemilius Macer, a jurist of the third century, whose work De Appellationibus ("On Appeals") is quoted as saying:
si calculi error in sententia esse dicatur, appellare necesse non est: veluti si iudex ita pronuntiaverit: "cum constet titium seio ex illa specie quinquaginta, item ex illa specie viginti quinque debere, idcirco lucium titium seio centum condemno": nam quoniam error computationis est, nec appellare necesse est et citra provocationem corrigitur. sed et si huius quaestionis iudex sententiam centum confirmaverit, si quidem ideo, quod quinquaginta et viginti quinque fieri centum putaverit, adhuc idem error computationis est nec appellare necesse est: si vero ideo, quoniam et alias species viginti quinque fuisse dixerit, appellationi locus est.
Translation from Samuel P. Scott, 1932:
If an error in the calculation is alleged to exist in the decision, it is not necessary to appeal, for instance, if the judge decides as follows: "As it is proved that Titius owes Seius fifty sesterces for such-and-such an article, and also twenty-five for another; therefore I hold that Lucius Titius shall pay Seius a hundred sesterces;" because, as the mistake is one of computation, it is not necessary to appeal, and it can be corrected without doing so. If, however, the judge of this question should render a decision for a hundred sesterces, for the reason that he thought that fifty and twenty-five made a hundred, still, the same mistake is one of computation, and it is not necessary to appeal. But when the judge decides that there is another sum of twenty-five sesterces due, there will be ground for appeal.
Another maxim for this situation is error calculi non nocet, an error in calculation does no harm. In the Corpus Iuris Civilis, 2.5, a similar formulation is attributed to a ruling from the co-Emperors Diocletian and Maximian, so the same rough time as Macer: errorem calculi ... veritati non adferre praeiudicum, an error in calculation does not prejudice the truth. The "non nocet" phrase can be found in much later sources, such as the 14th century commentary by Baldus de Ubaldis on Justinian. I think that this is a more precise and common term than iudex non calculat for that idea.
The cited law is that a judge can correct their own calculational error, to make the outcome accord with the just outcome they had intended. The form iudex non calculat matches this idea for saying that the arithmetical working is not primary, but rather something that is deduced from the decision. This is also a feature of modern law systems, wherein obvious mistakes in court orders can be corrected by the court itself, on application.
For the secondary meaning, assessing the strength of arguments rather than their quantity, this is so commonplace that it's hard to identify any particular source. It's certainly an idea we find in Cicero's rhetoric. His Orator, for example, has:
The stress of an argument does not always, and in every cause, depend upon similar topics. He will, therefore, exercise his judgment; and not only discover what may be said, but thoroughly examine the force of it. ... An Orator, therefore, should carefully examine each, that he may be able to select with propriety. Otherwise, how can he enlarge upon those which are most pertinent, and dwell upon such as more particularly affect his cause?
Cicero is suggesting that we should choose and hone our strongest arguments, not just reel off everything we can think of. Inclusion of weak or irrelevant material may make the overall case weaker, as well as trying the patience of the listener. I've heard similar sentiments from contemporary judges, so it's clearly something we've been struggling with for a couple of thousand years and counting.
The phrase "iudex non calculat" does not appear to have much current live use: perhaps mainly jurists disclaiming any interest in figures, using the phrase slightly ironically.
