By exporting the ability to map user address and inserting PTEs
representing their backing pages into the GTT, we can exploit UMA in order
to utilize normal application data as a texture source or even as a
render target (depending upon the capabilities of the chipset). This has
a number of uses, with zero-copy downloads to the GPU and efficient
readback making the intermixed streaming of CPU and GPU operations
fairly efficient. This ability has many widespread implications from
faster rendering of client-side software rasterisers (chromium),
mitigation of stalls due to read back (firefox) and to faster pipelining
of texture data (such as pixel buffer objects in GL or data blobs in CL).

v2: Compile with CONFIG_MMU_NOTIFIER
v3: We can sleep while performing invalidate-range, which we can utilise
to drop our page references prior to the kernel manipulating the vma
(for either discard or cloning) and so protect normal users.
v4: Only run the invalidate notifier if the range intercepts the bo.
v5: Prevent userspace from attempting to GTT mmap non-page aligned buffers
v6: Recheck after reacquire mutex for lost mmu.
v7: Fix implicit padding of ioctl struct by rounding to next 64bit boundary.
v8: Fix rebasing error after forwarding porting the back port.
v9: Limit the userptr to page aligned entries. We now expect userspace
    to handle all the offset-in-page adjustments itself.
v10: Prevent vma from being copied across fork to avoid issues with cow.
v11: Drop vma behaviour changes -- locking is nigh on impossible.
     Use a worker to load user pages to avoid lock inversions.
v12: Use get_task_mm()/mmput() for correct refcounting of mm.
v13: Use a worker to release the mmu_notifier to avoid lock inversion
v14: Decouple mmu_notifier from struct_mutex using a custom mmu_notifer
     with its own locking and tree of objects for each mm/mmu_notifier.
v15: Prevent overlapping userptr objects, and invalidate all objects
     within the mmu_notifier range
v16: Fix a typo for iterating over multiple objects in the range and
     rearrange error path to destroy the mmu_notifier locklessly.
     Also close a race between invalidate_range and the get_pages_worker.
v17: Close a race between get_pages_worker/invalidate_range and fresh
     allocations of the same userptr range - and notice that
     struct_mutex was presumed to be held when during creation it wasn't.
v18: Sigh. Fix the refactor of st_set_pages() to allocate enough memory
     for the struct sg_table and to clear it before reporting an error.
v19: Always error out on read-only userptr requests as we don't have the
     hardware infrastructure to support them at the moment.
v20: Refuse to implement read-only support until we have the required
     infrastructure - but reserve the bit in flags for future use.
v21: use_mm() is not required for get_user_pages(). It is only meant to
     be used to fix up the kernel thread's current->mm for use with
     copy_user().
v22: Use sg_alloc_table_from_pages for that chunky feeling
v23: Export a function for sanity checking dma-buf rather than encode
     userptr details elsewhere, and clean up comments based on
     suggestions by Bradley.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: "Gong, Zhipeng" <zhipeng.gong@intel.com>
Cc: Akash Goel <akash.goel@intel.com>
Cc: "Volkin, Bradley D" <bradley.d.volkin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Reviewed-by: Brad Volkin <bradley.d.volkin@intel.com>
[danvet: Frob ioctl allocation to pick the next one - will cause a bit
of fuss with create2 apparently, but such are the rules.]
[danvet2: oops, forgot to git add after manual patch application]
[danvet3: Appease sparse.]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Bit 31 in GEN6_PMINTRMSK is not an interrupt disable bit with gen8.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Otherwise, we do a NULL pointer dereference.

I've seen this happen while handling an error in
i915_gem_object_pin_to_display_plane():

If i915_gem_object_set_cache_level() fails, we call is_pin_display()
to handle the error. At this point, the object is still not pinned
to GGTT and maybe not even bound, so we have to check before we
dereference its GGTT vma.

The IGT kms_flip/bo-too-big tests for this bug.

v2: Chris Wilson says restoring the old value is easier, but that
is_pin_display is useful as a theory of operation. Take the solomonic
decision: at least this way is_pin_display is a little more robust
(until Chris can kill it off).

v3: Chris suggests the WARN in i915_gem_obj_to_ggtt has outlived its
usefulness: add a reminder to remove it.

Issue: VIZ-3772
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Testcase: igt/kms_flip/bo-too-big
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

All the callbacks are gone now.

Reviewed-by: Imre Deak <imre.deak@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Looking at our current dsi driver I note that:
- We don't have any slave driver right now.
- There's zero support for the hardware state readout and cross check
  code.
- All the modeset state seems to be tracked in the intel_dsi structure
  instead of the pipe config.

Given all that I can't properly audit the dsi ->mode_set callback. So
just do it as the first thing in the ->pre_pll_enable hook and hope
for the best.

Reviewed-by: Imre Deak <imre.deak@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

A bit more care required here since there are some very few things
between the call to encoder->mode_set and encoder->pre_enable. But
they're either book-keeping or only matter for the vga port on the
pch. So of no concern.

Note that with the new sequence we write the infoframes after
selecting the clock source, but that shouldn't matter. I've simply
opted for this to have simpler code.

Reviewed-by: Imre Deak <imre.deak@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

All the hard work was already done, only thing left to do is remove
the empty callback. And a now rather misleading comment I've spotted
while reading through code.

Reviewed-by: Imre Deak <imre.deak@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Similar to dp the only thing we do is call intel_write_eld and prepare
a bit of state for the enable hooks. The only difference is that we
write that to the hardware instead of keeping track of it somewhere in
software.

Still we can just move all this to the very first enable hook.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Those functions are only used on vlv platforms, so no need to check.
Especially if we're not too consistent about it.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

With all the preceding refactoring the dp mode_set callback only
computes a bit of state (all derived from the pipe config) and also
writes the eld. As long as we do that before we enable the audio bit
or depend upon the correct value in intel_dp->DP we'll be fine.

No other hw state is touched.

We therefore only need to check that clearing intel_dp->DP is save.
Which it is since when we re-enable we already mask out all the bits
the link training code sets. And we need to keep on doing that so that
the re-train loop walking over pre-emph/voltage-swing values still
works properly.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Only ilk/snb/ivb need the port A pll setup, so move it to the
pre_enable hook for those platforms. We can savely do this since on
those platforms there's nothing that touches the hardware between the
encoder->mode_set and the encoder->pre_enable calls.

Also add a comment that port A is ilk+ only.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Including state readout and cross-checking. This allows us to get rid
of crtc->eld_vld on hsw+. It also means that fastboot will be unhappy
if the BIOS hasn't set up the audio routing like we want it too.

Wrt fastboot and external screens I see a few options:
- Don't.
- Try to fix up eld, infoframes and audio settings after the fact. But
  that means some pretty extensive reworking of our code which
  currently does all this while the pipe/port is still off.

I won't bother with converting SDVO over to this because the audio
support for SDVO is very lacking:
- We don't update the eld.
- We don't update the audio state on the sdvo encoder.
- We don't check whether the platform can even feed audio to the sdvo
  encoder.

I've converted hdmi, dp & ddi all in one go since ddi needs both hdmi
and dp converted and so doing it step-by-step would have required a
few intermediate hacks.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

There's no need to check whether audio is enabled (which for ddi ports
is done through the crtc->eld_vld flag) since at the cost of a
potentially unecessary register rmw cycle we can unconditionally do
this.

Note that the edp check is just paranoia since we won't ever call the
write_eld function for an edp panel.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

This way we can rely on the state cross-checker to have a bit
assurance that we'll get it right.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

At least on those platforms which have a simple bit and don't rely
on the fully programmable CSC unit to do this.

Note that with the current code this includes CHV, but I guess that
platform will match BYT.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

We in the pre_enable hook we should only rely on the pipe config and
not on some other state set through properties or detect functions.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Also add state readout and cross-check support. The only invasive change
is wiring up the new flag to the ->set_infoframes callbacks.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

For compliance we really should be sending NULL infoframes always
when we detect a hdmi capable monitor. Also remove the now redudant
setting for the has_audio case and enforce that audio is only
possible with a hdmi sink.

Reviewed-by: Naresh Kumar Kachhi <naresh.kumar.kachhi@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Everything should be in place so enable rc6/rps for bdw.

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

by correctly displaying result and requested.

Suggested-by: Imre Deak <imre.deak@intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

In BDW, Apart from unmasking up/down threshold interrupts. we need
to umask bit 32 of PM_INTRMASK to route interrupts to target via Display
Interface.

v2: Add (1<<31) mask (Ville)

v3: Add Gen check for the mask (ville)

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Almost all of it is reusable from the existing code. The primary
difference is we need to do even less in the interrupt handler, since
interrupts are not shared in the same way.

The patch is mostly a copy-paste of the existing snb+ code, with updates
to the relevant parts requiring changes to the interrupt handling. As
such it /should/ be relatively trivial. It's highly likely that I missed
some places where I need a gen8 version of the PM interrupts, but it has
become invisible to me by now.

This patch could probably be split into adding the new functions,
followed by actually handling the interrupts. Since the code is
currently disabled (and broken) I think the patch stands better by
itself.

v2: Move the commit about not touching the ringbuffer interrupt to the
snb_* function where it belongs (Rodrigo)

v3: Rebased on Paulo's runtime PM changes

v4: Not well validated, but rebase on
commit 730488b2


Author: Paulo Zanoni <paulo.r.zanoni@intel.com>
Date:   Fri Mar 7 20:12:32 2014 -0300

    drm/i915: kill dev_priv->pm.regsave

v5: Rebased on latest code base. (Deepak)

v6: Remove conflict markers, Unnecessary empty line and use right
IIR interrupt (Ville)

v7: mask modified without rmw (Ville Syrjälä)

Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

If we dont have semaphores enabled, we allocate 4
dwords for signalling. But end up emitting more regardless.

Fix this by bailing out early if semaphores are not enabled.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=78274
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=78283
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Added as generic parameters which will be initialized in the panel
driver and are specific to panels.

Backlight delays have also kept as placeholders and will be used used
once we have MIPI backlight enabling support

Signed-off-by: Shobhit Kumar <shobhit.kumar@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: Shobhit Kumar <shobhit.kumar@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

In VBT fields operation mode is 0 for Video mode and 1 for command mode.
This field will be directly used as is in generic panel driver. So
adjust accordingly.

Signed-off-by: Shobhit Kumar <shobhit.kumar@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

These are generated with intel-gpu-tools/tools/null_state_gen

v2: Don't use header file for states (Daniel Vetter)

v3: Proper URB state size for gen8/GT3 (Damien Lespiau)

Tested-by: Kristen Carlson Accardi <kristen@linux.intel.com> (v1)
Tested-by: Oscar Mateo <oscar.mateo@intel.com> (v2)
Acked-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

HW guys say that it is not a cool idea to let device
go into rc6 without proper 3d pipeline state.

For each new uninitialized context, generate a
valid null render state to be run on context
creation.

This patch introduces a skeleton with empty states.

v2: - No need to vmap (Chris Wilson)
    - use .c files for state (Daniel Vetter)
    - no need to flush as i915_add_request does it
    - remove parameter for batch alloc size
    - don't wait for the init (Ben Widawsky)

v3: - move to cpu/gpu (Chris Wilson)

Tested-by: Kristen Carlson Accardi <kristen@linux.intel.com> (v1)
Tested-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Otherwise we end up tearing down fences when e.g. the client quits
way too early. Might or might not fix a fence pin_count BUG Ville has
reported.

Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Userspace can currently provoke this when e.g. trying to use a pinned
scanout as a cursor or overlay target. Later on that might lead to
some fun fence pin count mayhem.

Spurred by Ville's report that something goes wrong here and
originally I've thought that this might slip through the pwrite gtt
fastpath. But that one checks of obj tiling, so should be ok.

But one thing that _does_ blow up is the vma unbinding with more than
one address space. The next patch will fix this.

v2: Use a WARN_ON - Chris pointed out that we already catch all cases
so userspace can't provoke this like I've originally feared.

While reviewing relevant code I've noticed a pile of DRM_ERROR in the
overlay&cursor code which are all triggerable by userspace. Tune them
down while at it.

v3: Split out the DRM_ERROR->DRM_DEBUG_KMS change into a separate patch,
as requested by Chris.

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Tested-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Noticed while playing with coccinelle.

Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

During initial probing of the modes to assign to the fbdev console, we
use the CRTC and connector ids. These are much harder for us to
understand than if we used their actual names (or pipe in the CRTC
case). Similarly, we want to manually print the mode size rather than
rely on mode->name being set.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Patch done using the following semantic patch (thanks Daniel for the
help!)

  @@
  iterator name list_for_each_entry;
  iterator name for_each_crtc;
  struct drm_crtc * crtc;
  struct drm_device * dev;
  @@
  -list_for_each_entry(crtc,&dev->mode_config.crtc_list, head) {
  +for_each_crtc(dev,crtc) {
  	...
  }

Followed by a couple of fixups by hand (that spatch doesn't match the
cases where list_for_each_entry() is not followed by a set of '{', '}',
but I couldn't figure out a way to leave the '{' out of the iterator
match).

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Generated using the semantic patch:

  @@
  iterator name list_for_each_entry;
  iterator name for_each_intel_crtc;
  struct intel_crtc * crtc;
  struct drm_device * dev;
  @@
  -list_for_each_entry(crtc,&dev->mode_config.crtc_list,...) {
  +for_each_intel_crtc(dev,crtc) {
	...
  }

Followed by a couple of fixups by hand (that spatch doesn't match the
cases where list_for_each_entry() is not followed by a set of '{', '}',
but I couldn't figure out a way to leave the '{' out of the iterator
match).

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Fed up with having that long list_for_each_entry() invocation?

Use for_each_intel_crtc()!

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
[danvet: Squash in patch that exported ilk_wm_max_level.]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

That's not necessary and makes the code not as neat as it could be.

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Somehow UXA submits a completely bogus DR4 value since essentially
forever. It was originally introduced in

commit bade7d7d2505a10a8a7d24b084aff9742e2d6d64
Author: Eric Anholt <eric@anholt.net>
Date:   Fri Jun 6 14:03:25 2008 -0700

    Use the DRM for submitting batchbuffers when available.

and dutifully copied around ever since. Since we want to keep the
general dirt catching around just special case the UXA value.

This regression was introduced in

commit 9cb34664


Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Thu Apr 24 08:09:11 2014 +0200

    drm/i915: Catch dirt in unused execbuffer fields

Comment from Chris' review:

"To be fair, it is a sensible value if one supposes a Region style API to
cliprects. Under that API, DR[14] define the extents of the clip region,
and ((0,0), (0,0)) [DR1==DR4==0] would mean all clipped, do not draw
anything."

v2: Pimp commit message a bit and remove the double space.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=78494
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jörg Otte <jrg.otte@gmail.com>
Acked-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

The fence pin count should always be <= the bo pin count. If that's
not the case then we have a funny problem and are leaking references
somewhere.

Which means we can catch fence pin leaks by checking for the same
upper limit as we do for the bo pin count. Inspired by a discussion
with Ville about a fence leak igt testcase.

v2: Also check for fence->pin_count <= ggtt_vma->pin_count, since that
might catch a leak even quicker. Also de-inline them, they're getting
too big.

v3: Don't separately check for MAX_PIN_COUNT since the > vma->pin_count
check will catch that already (Chris).

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>