Broadcom BCM63xx DSL SoCs utilize BMIPS CPUs, and as such are required
to perform a read-ahead cache flush after a DMA transfer. Utilize
asm/bmips.h to provide a plat_post_dma_flush_hook, and
mach-generic/dma-coherence.h for everything else.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: cernekee@gmail.com
Cc: jogo@openwrt.org
Patchwork: https://patchwork.linux-mips.org/patch/9726/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Instead of having platforms to copy the entirety of
mach-generic/dma-coherence.h, check whether these platforms have already
defined a plat_post_dma_flush hook, and if not, provide an inline stub.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: cernekee@gmail.com
Cc: jogo@openwrt.org
Patchwork: https://patchwork.linux-mips.org/patch/9725/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

arch/mips/include/asm/mach-bmips/dma-coherence.h contains the
plat_post_dma_flush implementation which is not specific to mach-bmips,
but required for all BMIPS-based systems.

Move plat_post_dma_flush to arch/mips/include/asm/bmips.h, rename it to
bmips_post_dma_flush such that other platforms like bcm63xx can utilize
it.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: cernekee@gmail.com
Cc: jogo@openwrt.org
Patchwork: https://patchwork.linux-mips.org/patch/9724/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Introduced in 34c2f668

 (MIPS: microMIPS:
Add unaligned access support.)

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Following the arrangement for processors that wire FPU exceptions to the
FPE CPU exception handle the case where no FPU is in use -- which for
DECstation systems will only ever happen when the "nofpu" kernel option
has been used -- do not register the FPU interrupt in such a case
either.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9714/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Implement a cheap way to count FPU interrupts for R2k/R3k DECstation
systems.  Do this manually in handcoded assembly, rather than calling
`kstat_incr_irq_this_cpu' that would require setting up a stack frame
and a lot of redirection.  This is not going to be a problem because the
FPU interrupt is local to the CPU and also there is one CPU only anyway.

So at bootstrap determine the address of the correct location within
`struct irq_desc', and then only refer to it directly in the interrupt
handler.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9713/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Factor out FPU feature probing, mainly to remove code duplication from
`fpu_disable'.  No functional change although shuffle some code to avoid
forward references.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9712/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Define the central place the default FCSR value is set from, initialised
in `cpu_probe'.  Determine the FCSR mask applied to values written to
the register with CTC1 in the full emulation mode and via ptrace(2),
according to the ISA level of processor hardware or the writability of
bits 31:18 if actual FPU hardware is used.

Software may rely on FCSR bits whose functions our emulator does not
implement, so it should not allow them to be set or software may get
confused.  For ptrace(2) it's just sanity.

[ralf@linux-mips.org: Fixed double inclusion of <asm/current.h>.]

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9711/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

The ABS.fmt and NEG.fmt instructions have been specified as arithmetic
in the MIPS architecture, which in particular implies handling NaN data
in the usual way with qNaN bit patterns propagated unchanged and sNaN
bit patterns signalling the usual IEEE 754 Invalid Operation exception
and quieted by default.

A series of changes applied over time to our implementation:

c5033d78 [MIPS] ieee754[sd]p_neg workaround
cea2be44

 MIPS: Fix abs.[sd] and neg.[sd] emulation for NaN operands

has led to the current situation where the sign bit is updated according
to the operation requested even for NaN inputs.  This is according to
these commits a workaround so that broken binaries produced by GCC
disregarding the properties of these instructions have a chance to work.

For sNaN inputs this remains within IEEE Std 754 as the standard leaves
the choice of output qNaN bit patterns produced under the default
Invalid Operation exception handling for individual sNaN input bit
patterns to implementer's discretion, even though it still recommends as
much NaN input information to be preserved in NaN outputs.

For qNaN inputs however it violates the standard as it requires a qNaN
input bit patterns to propagate unchanged to output.

This is also unlike real MIPS FPU hardware behaves where sNaN and/or
qNaN processing has been fully implemented with no Unimplemented
Operation exception signalled.  Such hardware propagates any input qNaN
bit pattern unchanged.  It also quiets any input sNaN bit pattern in an
implementer-specific manner, for example the MIPS 74Kf processor returns
the default qNaN pattern with the sign bit always clear and the Broadcom
SB-1 and BMIPS5000 processors propagate the input sNaN bit pattern with
the sign bit unchanged and the quiet bit first cleared in the trailing
significand field and then the next lower bit set if clearing the quiet
bit left the field with no other bit set.

Especially the latter observation indicates the limited usefulness of
the workaround as it will cover many hardware configurations, but not
all of them, only making it harder to discover such broken binaries that
need to be recompiled with GCC told to avoid the use of ABS.fmt and
NEG.fmt instructions where non-arithmetic semantics is required by the
algorithm used.

Revert the damage done by the series of changes then, and take the
opportunity to simplify implementation by calling `ieee754dp_sub' and
`ieee754dp_add' as required and also the rounding mode set towards -Inf
temporarily so that the sign of 0 is correctly handled.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9710/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Define IEEE 754-2008 feature control bits: FIR.HAS2008, FCSR.ABS2008 and
FCSR.NAN2008, and update the `_ieee754_csr' structure accordingly.

For completeness define FIR.UFRP too.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9709/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Implement the FCCR, FEXR and FENR "shadow" FPU registers for the
architecture levels that include them, for the CFC1 and CTC1
instructions in the full emulation mode.

For completeness add macros for the CP1 UFR and UNFR registers too, no
actual implementation though.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9708/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Implement FIR feature flags in the FPU emulator according to features
supported and architecture level requirements.  The W, L and F64 bits
have only been added at level #2 even though the features they refer to
were also included with the MIPS64r1 ISA and the W fixed-point format
also with the MIPS32r1 ISA.

This is only relevant for the full emulation mode and the emulated CFC1
instruction as well as ptrace(2) accesses.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9707/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Correct an ISA level determination problem introduced with 8b8aa636


[MIPS: kernel: cpu-probe.c: Add support for MIPS R6], reverting explicit
masking against individual `MIPS_CPU_ISA_*' macros in FPU feature
determination.

Feature macros such as `cpu_has_mips_r' cannot be used here, because
they operate on CPU #0 and we want to refer to the current CPU instead.
They cannot be used for masking against the current CPU either because
they mask against CPU #0 too, e.g.:

# define cpu_has_mips32r1	(cpu_data[0].isa_level & MIPS_CPU_ISA_M32R1)

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9706/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Rework `process_fpemu_return' and move IEEE 754 exception interpretation
there, from `do_fpe'.  Record the cause bits set in FCSR before they are
cleared and pass them through to `process_fpemu_return' so as to set
`si_code' correctly too for SIGFPE signals sent from emulation rather
than those issued by hardware with the FPE processor exception only.

For simplicity `mipsr2_decoder' assumes `*fcr31' has been preinitialised
and only sets it to anything if an FPU instruction has been emulated,
which in turn is the only case SIGFPE can be issued for here.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9705/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Clear any FCSR cause bits recorded in the saved FPU context after
emulation in all cases rather than in `do_fpe' only, so that any
unmasked IEEE 754 exception left from emulation does not cause a fatal
kernel-mode FPE hardware exception with the CTC1 instruction used by the
kernel to subsequently restore FCSR hardware from the saved FPU context.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9704/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Respect the FCSR exception mask when interpreting the IEEE 754 exception
condition to report with SIGFPE in `si_code', so as not to use one that
has been masked where a different one set in parallel caused the FPE
hardware exception to trigger.  As per the IEEE Std 754 the Inexact
exception can happen together with Overflow or Underflow.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9703/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Complement 593d33fe

 [MIPS: math-emu: Move various objects into an ar
library.] and also move sp_tlong.o, sp_flong.o, dp_tlong.o, and
dp_flong.o into an `ar' library.  These objects implement long
fixed-point format support that can be omitted from MIPS I, MIPS II and
MIPS32r1 configurations.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9702/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Restore EPC at the branch whose delay slot is emulated if the delay-slot
instruction signals.  This is so that code in `fpu_emulator_cop1Handler'
does not see EPC having advanced and mistakenly successfully resume
userland execution from the location at the branch target in that case.
Restoring EPC guarantees an immediate exit from the emulation loop and
if EPC hasn't advanced at all since entering the loop, also issuing the
signal reported by the delay-slot instruction.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9701/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Correct ISA requirements for floating-point instructions:

* the CU3 exception signifies a real COP3 instruction in MIPS I & II,

* the BC1FL and BC1TL instructions are not supported in MIPS I,

* the SQRT.fmt instructions are indeed supported in MIPS II,

* the LDC1 and SDC1 instructions are indeed supported in MIPS32r1,

* the CEIL.W.fmt, FLOOR.W.fmt, ROUND.W.fmt and TRUNC.W.fmt instructions
  are indeed supported in MIPS32,

* the CVT.L.fmt and CVT.fmt.L instructions are indeed supported in
  MIPS32r2 and MIPS32r6,

* the CEIL.L.fmt, FLOOR.L.fmt, ROUND.L.fmt and TRUNC.L.fmt instructions
  are indeed supported in MIPS32r2 and MIPS32r6,

* the RSQRT.fmt and RECIP.fmt instructions are indeed supported in
  MIPS64r1,

Also simplify conditionals for MIPS III and MIPS IV FPU instructions and
the handling of the MOVCI minor opcode.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9700/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Implement the correct ordering of individual floating-point registers
within double-precision register pairs for the MIPS I FP context, as
required by our FP emulation code and expected by userland talking via
ptrace(2).  Use L.D and S.D assembly macros that do the right thing like
LDC1 and SDC1 from MIPS II up, avoiding the need to mess up with
endianness conditionals.

This in particular fixes the handling of denormals and NaN generation in
Unimplemented Operation emulation traps.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9699/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Correct a cache coherency regression introduced with be1664c4 [Another
round of fixes for the fp emulator.] for the emulation frame used in
delay-slot emulation.

Two instructions are copied into the frame and as from the commit
referred a cache synchronisation call is made for the second instruction
aka `badinst' of the two only.  The `flush_cache_sigtramp' interface is
reused that guarantees that synchronisation will be made for 8 bytes or
2 instructions starting from the address requested, although if cache
lines are wider then a larger area may be synchronised.

Change the call to point to the first of the two instructions aka `emul'
instead, removing unpredictable behaviour resulting from cache
incoherency.

This bug only ever manifested itself on systems implementing 4-byte
cache lines, typically MIPS I systems, causing all kinds of weirdness.
This is because the sequence of two instructions starting from `emul' is
8-byte aligned and for 8-byte or wider cache lines the line synchronised
will span both, so the vast majority of systems have escaped unharmed.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9698/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Do not lose the other half of the BREAK code where there is an upper
half.  This is so that e.g. `BREAK 7, 7' is not interpreted as a divide
by zero trap, while `BREAK 0, 7' or `BREAK 7, 0' still are.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9697/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Add the missing microMIPS BREAK16 instruction code interpretation and
reshape code removing instruction fetching duplication and the separate
call to `do_trap_or_bp' in the MIPS16 path.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9696/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Correct the interpretation of the immediate MIPS16 BREAK instruction
code embedded in the instruction word across bits 10:5 rather than 11:6
as current code implies, fixing the interpretation of integer overflow
and divide by zero traps.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9695/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

The `cpu_has_fpu' feature flag must not be hardcoded to 1 or the `nofpu'
kernel option will be ignored.  Remove any such overrides and add a
cautionary note.  Hardcoding to 0 is fine for FPU-less platforms.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9694/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

The `ieee754sp_isnan' and `ieee754dp_isnan' NaN classifiers are now no
longer externally referred, remove their header prototypes and make them
local to the two only respective places still making use of them.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9693/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Rewrite qNaN handling in `ieee754sp_fdp' using the `ieee754_class_nan'
helper recently added, removing the external call to `ieee754sp_isnan'
and reducing the size of code by 16 instructions or 64 bytes.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9692/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

None of the comparison helpers in ieee754.h is used, remove them.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9691/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Remove a redundant call to `ieee754_setandtestcx' in `ieee754sp_cmp' and
`ieee754dp_cmp'.  The IEEE 754 exception requested will have already
been set by a call to `ieee754_setcx' immediately above, because `sig'
has to be non-zero to reach here, and the comparison result returned
will be 0 regardless of the result from the call.  Simplify the return
expression remaining.  All this reducing the size of code by 16 and 12
instructions or 64 and 48 bytes respectively.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9690/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

We have the input operands already classified in `ieee754sp_cmp' and
`ieee754dp_cmp' comparison operations, so use the class obtained to tell
NaNs and numbers apart rather than classifying inputs again for this
purpose, reducing the size of code by 24 and 40 instructions or 96 and
160 bytes respectively.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9689/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Revert the changes made by commit fdffbafb [Lots of FPU bug fixes from
Kjeld Borch Egevang.] to `ieee754sp_nanxcpt' and `ieee754dp_nanxcpt'
sNaN quieting handlers and their callers so that sNaN processing is done
within the handlers againg.  Pass the sNaN causing an IEEE 754 invalid
operation exception down to the relevant handler.  Pass the sNaN in `fs'
where two sNaNs are supplied to a binary operation.

Set the Invalid Operation FCSR exception bits in the quieting handlers
rather than at their call sites throughout.  Make the handlers exclusive
for sNaN processing.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9688/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Don't call the `ieee754sp_nanxcpt' and `ieee754dp_nanxcpt' sNaN quieting
handlers for a qNaN supplied to floating-point format conversions or
SQRT.S/SQRT.D instructions, or for a qNaN produced out of a negative
operand supplied to SQRT.S/SQRT.D instructions.  Return the qNaN right
away in these cases.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9687/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9686/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Commit fdffbafb [Lots of FPU bug fixes from Kjeld Borch Egevang.]
replaced the two single `ieee754sp_nanxcpt' and `ieee754dp_nanxcpt'
places, where sNaN quieting used to happen for single and double
floating-point operations respectively, with individual qNaN
instantiations across all the call sites instead.  It also made most of
these two functions dead code as where called on a qNaN they return
right away.

To revert the damage and make sNaN quieting uniform again first rewrite
`ieee754sp_nanxcpt' and `ieee754dp_nanxcpt' to do the same quieting all
the call sites do, that is return the default qNaN encoding for all
input sNaN values; never propagate any sNaN payload bits from its
trailing significand field.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9685/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Replace a hardcoded numeric bitmask for FCSR cause bits with
`FPU_CSR_ALL_X' in `__build_clear_fpe'.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9684/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Changes applied to `do_cpu' over time reduced the use of the SIGILL
issued with `force_sig' at the end to a single CU3 case only in the
switch statement there.  Move that `force_sig' call over to right where
required then and toss out the pile of gotos now not needed to skip over
the call, replacing them with regular breaks out of the switch.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9683/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Move CFC1/CTC1 emulation code to separate functions to avoid excessive
indentation in forthcoming changes.  Adjust formatting in a minor way
and remove extraneous round brackets.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9682/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

GCC is smart enough to substitute the final result for FLS calculations
as implemented in the fallback C code we have in `__fls' and `fls'
applied to constant values.  The presence of inline asm defeats the
compiler though, forcing it to emit extraneous CLZ/DCLZ calculation for
processors that support these instructions.

Use `__builtin_constant_p' then to avoid inline asm altogether for
constants.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9681/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>