lib/math/rational.c: fix possible incorrect result from rational fractions helper

In some cases the previous algorithm would not return the closest
approximation.  This would happen when a semi-convergent was the
closest, as the previous algorithm would only consider convergents.

As an example, consider an initial value of 5/4, and trying to find the
closest approximation with a maximum of 4 for numerator and denominator.
The previous algorithm would return 1/1 as the closest approximation,
while this version will return the correct answer of 4/3.

To do this, the main loop performs effectively the same operations as it
did before.  It must now keep track of the last three approximations,
n2/d2 ..  n0/d0, while before it only needed the last two.

If an exact answer is not found, the algorithm will now calculate the
best semi-convergent term, t, which is a single expression with two
divisions:

    min((max_numerator - n0) / n1, (max_denominator - d0) / d1)

This will be used if it is better than previou...